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ABSTRACT 

The  problem  of  macro-scheduling  in  systems  with  a  variety  of  processors  so 
as  to  achieve  high  productivity  is  investigated.  In  order  to  increase  productivity, 
an  attempt  is  made  at  every  scheduling  interval  to  reach  system  balance  -  a 
state  in  which  processor  utilizations  are  at  prespecified  levels.  An  initial 
approach  is  to  implement  traditional  techniques  for  balancing  the  system.  A 
simulation  study  of  algorithms  designed  to  maximize  processor  utilizations  indi¬ 
cates  an  improvement  in  the  average  of  processor  utilizations  by  up  to  15  per¬ 
centage  points  over  a  neutral  algorithm.  None  of  the  examined  macro- 
schedulers  however  proved  to  be  consistently  superior  to  the  others. 

A  drawback  of  current  productivity  oriented  schedulers  is  that  there  is 
inadequate  account  of  the  effect  of  interactions  among  active  jobs  on  processor 
utilizations.  A  new  approach  to  the  problem  is  to  consider  an  analytic  model  of 
the  system,  in  which  processor  utilizations  can  be  expressed  as  functions  of  the 
service  demands  of  active  jobs.  To  this  end,  mean  value  analysis  is  applied  to  a 
separable,  Markovian,  load-independent  model  of  the  system.  Four  results  of 
this  analysis  are:  1.  a  macro-scheduling  algorithm,  OPT,  which  is  optimal  with 
respect  to  the  model  assumptions,  2.  an  approximate  algorithm  which  is  compu¬ 
tationally  more  efficient  than  the  optimal  for  cases  of  high  multiprogramming 
levels,  3.  a  framework  for  analysis  and  comparison  of  other  heuristic  macro¬ 
schedulers,  and  4.  a  proof  that  only  the  ratios  of  intrinsic  demands  of  a  job  are 
required  for  optimal  scheduling.  Comparative  analytic  studies  and  simulation 
studies  of  seven  balancing  macro-schedulers  are  described.  Simulations  suggest 
that  the  optimal  and  the  approximate  algorithms  for  separable  systems  are 
superior  to  other  algorithms,  evert  when  some  model  assumptions  are  violated. 


The  long-term  effect  of  repetitive  balancing  on  productivity  is  examined  under  a 
high  variance  workload.  In  a  typical  example,  the  elapsed-t.ime  required  to  pro¬ 
cess  a  given  set  of  jobs  under  OPT  is  reduced  by  9.7%  compared  to  the  elapsed- 
time  required  to  process  the  same  workload  under  a  neutral  scheduler.  For  this 
example,  95%  confidence  intervals  are  established.  The  results  indicate  that 
significant  gains  in  productivity  can  be  achieved  by  employing  balancing  macro¬ 


schedulers. 
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CHAPTER  ONE 


Introduction  and  Problem  Definition 

A  computer  system  consists  of  human,  software,  and  hardware  resources 
which  interact  to  provide  computational  and  data  storage  services.  The  perfor¬ 
mance  objectives  associated  with  the  use  of  a  computer  system  can  be  classified 
as  user-oriented  or  system-oriented.  A  user  wants  his  (her)  job  to  be  executed 
correctly,  inexpensively,  and  quickly.  In  addition  to  satisfying  individual  user 
requirements,  the  administrators  of  a  computer  system  are  also  interested  In 
high  productivity,  i.e.  in  processing  a  large  workload  within  a  short  time  interval, 
in  order  to  utilize  available  resources  fully  and  to  increase  revenues.  Thus,  the 
system-oriented  objectives  include  the  user-oriented  objectives  plus  the  objec¬ 
tive  of  high  productivity.  As  a  means  to  advance  the  user-oriented  objectives  as 
well  as  to  achieve  high  productivity,  fast  hardware  devices  with  large  storage 
capacity  have  been  invented.  Operating  systems  have  also  undergone  elaborate 
changes  to  enhance  the  potential  of  the  system.  New  generation  operating  sys¬ 
tems  are  capable  of  multiprocessing  -  allowing  a  number  of  central  processing 
devices  to  work  in  parallel,  and  multiprogramming  -  allowing  main  storage 
residence  and  concurrent  execution  of  more  than  one  job  at  a  time.  Some 
operating  systems  employ  virtual  memory  to  reduce  the  constraint  of  limited 
main  storage  by  allowing  a  job  to  be  executed  in  a  main  storage  region  which  is 
less  than  its  address  space  requirement. 

In  a  modern,  multiprogrammed  computer  system,  despite  the  advances  in 
hardware  and  software,  a  common  situation  arises  due  to  limited  resource 
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capacity  wherein  only  a  small  fraction  of  the  jobs  available  for  execution  is  con¬ 
currently  processed.  As  a  result,  macro-scheduling  ,  i.e.  deciding  on  the  order 
in  which  jobs  will  be  executed,  affects  both  the  high  productivity  objective  and 
the  user  objectives.  The  aim  of  this  thesis  is  the  determination,  of  a  macro- 
scheduling  algorithm  which  significantly  improves  system  productivity.  A 
macro-scheduler  whose  objective  is  to  increase  productivity  will  be  referred  to 
as  a  productivity  scheduler  or  as  a  productivity  algorithm. 

One  could  argue  that  with  the  decreasing  cost  of  hardware,  there  is  no 
benefit  in  installing  a  productivity  scheduler  since  an  increase  in  system  capa¬ 
city  can  be  obtained  by  adding  more  devices.  The  following  counter  arguments 
are  suggested: 

1.  The  cost  of  a  hardware  resource  such  as  a  central  processing  unit  (CPU)  or  a 
channel  is  by  no  means  negligible.  Poor  selection  of  jobs  for  the  service  set 
might  result  in  different  resources  becoming  congested  at  different  times.  In 
order  to  cope  with  high  contention  at  various  resources,  a  number  of  devices  will 
have  to  be  added  or  upgraded,  incurring  a  large  expenditure. 

2.  The  use  of  computer  systems  increases  as  hardware  costs  decrease  and  as 
systems  become  larger.  Consequently,  contention  for  resources  is  unlikely  to 
vanish,  making  the  optimization  of  resource  usage  a  valid  goal. 

In  the  remainder  of  the  introduction  the  problem  of  macro-scheduling  for  high 
productivity  is  discussed  in  greater  detail. 

The  macro-scheduler  is  the  part  of  an  operating  system  responsible  for  the 
make-up  of  the  service  set *  which  is  the  set  of  jobs  residing  in  main  storage. 
When  a  job  is  chosen  for  the  service  set,  part  of  the  job’s  address  space  is 
swapped  into  main  storage.  In  early  operating  systems,  the  entire  address  space 

*  The  service  set  is  also  known  as  the  active  mix,  the  multiprogramming  mix.,  or  the  resident 
set. 
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had  to  be  swapped  in.  More  advanced  systems  allowed  the  user  to  specify  parts 
of  the  address  space  to  be  overiayed  in  maim  storage.  In  many  modern  operat¬ 
ing  systems  a  virtual  memory  is  employed  with  mechanisms  to  estimate  the  por¬ 
tion  of  the  address  space  necessary  to  be  loaded  into  main  storage  and  to  facili¬ 
tate  the  loading  of  this  portion. 

At  some  instants  called  invocation  times,  the  operating  system  calls  iipon 
the  macro-scheduler  to  select  jobs  to  be  loaded  into  main  storage  and  to  be  pro¬ 
cessed.  During  a  scheduling  interval,  the  time  between  two  consecutive  invoca¬ 
tions,  no  new  jobs  are  loaded.  Micro-level  activities  are  performed  during  a 
scheduling  interval.  They  include  job  processing  and  micro-scheduling,  the 
determination  of  the  order  in  which  service  set  jobs  arc  processed. 

The  macro-scheduler  selects  a  job  from  the  inactive  set,  i.e.  from  the  set  of 
jobs  which  resides  in  secondary  storage  ready  to  be  executed.  In  preemptive 
systems  the  macro-scheduler  decides  what  jobs  to  swap  out  of,  as  well  as  into, 
main  storage;  jobs  can  be  added  to  or  deleted  from  the  service  set.  In 
noripreemplive  systems,  once  a  job  is  loaded,  il  remains  in  the  service  set  to 
completion. 

The  selection  of  jobs  for  the  service  set  influences  user-oriented  perfor¬ 
mance  measures  as  well  as  productivity.  For  example,  a  class  of  customers  can 
be  guaranteed  fast  response  time  with  consistently  preferential  treatment. 
Another  example  of  the  effect  the  choice  of  the  service  set  has  on  performance 
is  a  combination  of  active  jobs  causing  one  processor  to  become  heavily  utilized 
while  others  experience  low  utilizations.  With  demands  of  jobs  in  the  inactive  set 
being  equal  for  all  processors,  the  result  is  low  throughput  -  a  slowdown  in  the 
rate  at  which  jobs  are  completed  by  the  system. 

User-oriented  objectives  might  conflict  with  the  objective  of  high  produc¬ 
tivity.  A  major  problem  in  the  design  of  macro-schedulers  is  the  resolution  of 
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this  conflict.  Whereas  most  of  the  research  in  macro-scheduling  has  concen¬ 
trated  on  user-oriented  objectives,  very  little  is  known  of  productivity  algo¬ 
rithms  in  ■which  a  variety  of  resources  is  considered.  Tn  this  thesis  we  are  con¬ 
cerned  with  maximizing'  productivity.  Why  bother  optimizing  one  performance 
measure  if,  in  reality,  the  macro-scheduler  needs  to  reach  a  compromise 
between  conflicting  objectives?  Three  answers  are  suggested: 

1.  The  objectives  can  be  pursued  by  two  macro-scheduling  modules  invoked  in 
parallel.  Each  module  is  designed  to  optimize  one  objective,  and  to  compute  a 
recommended  value  for  each  of  the  candidates  for  addition  to  the  service  set. 
The  selection  of  a  job  for  the  service  set  is  based  on  a  combination  of  the  recom¬ 
mended  values  produced  by  the  two  schedulers. 

2.  The  objectives  can  be  pursued  by  two  macro-scheduling  modules  of  which  only 
one  is  invoked  at  a  time.  Each  module  is  designed  to  optimize  one  objective. 
Normally,  a  macro-scheduler  for  optimizing  the  user  objectives  is  used.  Only 
when  special  conditions  arise  will  the  operating  system  switch  to  invoke  the  pro¬ 
ductivity  algorithm.  Such  conditions  include  an  exceptionally  high  load  on  the 
system,  underutilization/overutilization  of  processors,  or  acceptable  user  per¬ 
formance  measures  combined  with  the  possibility  of  exercising  other  options. 

3.  The  objectives  are  not  always  equally  important.  In  some  situations  the  objec¬ 
tive  of  high  system  productivity  is  more  important  than  the  objective  of  fast 
response  time  for  individual  jobs.  For  instance,  consider  a  night  shift  at  a  com¬ 
puter  system-  in  a  bank.  A  typical  workload  consists  of  independent  jobs  which 
make  heavy  use  of  resources  such  as  daily  balance  calculation,  interest  update, 
report  generation,  and  system  maintenance.  Completion  times  of  individual  jobs 
during  the  night  are  insignificant,  as  long  as  the  entire  workload  is  processed  by 
the  end  of  the  shift.  In  order  to  ensure  processing  of  the  workload  within  the 
duration  of  the  shift,  a  productivity  scheduler  would  be  used. 
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Si  Lice  a  macro-scheduler  has.  to  provide  recommendations  for  the  next 
scheduling  interval,  without  a  priori  knowledge  of  new  jobs  which  may  arrive  at 
the  inactive  set  during  the  scheduling  interval,  we  make  the  macro-scheduler 
best-one-step.  In  a  best-one-step* **  algorithm  an  attempt  is  made  to  optimize  a 
performance  measure  for  the  next  scheduling  interval  only.  Optimization  over  a 
scheduling  interval  is  but  a  means  to  improve  productivity  over  much  longer 
intervals  such  as  a  shift,  a  day,  or  the  time  needed  to  execute  a  heavy  workload. 
Employment  of  a  best-one-step  algorithm  might  be  detrimental  in  the  long  run, 
because  optimization  of  a  performance  measure  over  short  intervals  might 
defer  execution  of  certain  jobs  whose  eventual  processing  would  result  in  long 
intervals  of  unacceptable  performance.  A  best-one-step  algorithm  is  considered 
benefitiai  only  if  it  improves  performance  in  the  long  run. 

Storage  optimization  will  not  be  included  among  the  objectives  of  the 
macro-scheduler.  Storage  optimization  is  connected  with  the  problem  of  choos¬ 
ing  an  appropriate  level  of  multiprogramming.  Operating  systems  usually 
employ  special  storage  management  modules  to  resolve  such  problems.  Storage 
constraints  will  be  implicitly  considered  by  limiting  the  number  of  jobs  in  the 
service  set. 

"We  are  now  in  a  position  to  pose,  in  a  precise  way,  the  question  addressed  in 
the  following  chapters.  The  system  under,  consideration  is  a  collection  of 
different  processors.  At  some  instant  a  job  can  join  the.  service  set.  The  size  of 
the  service  set  may,  or  may  not,  increase  as  a  result  of  the  additional  job.  The 
question  is  -  which  inactive  job  should  be  selected  for  the  service  set,  such  that 
its  activation  will  result  in  bringing  processor  utilization^*  as  close  as  possible 

to  prespecified  levels.  This  selection  must  also  result  in  a  shorter  elapsed-time 

*  The  best-one-step  algorithm  is  similar  to  the  greedy  algorithm  (Lawler  1978). 

**  Throughout  the  thesis,  unless  otherwise  specified,  processor  utilization  means  the  fraction 
of  a  measurement  interval  during  which  the  processor  was  busy  serving  jobs  only,  excluding 
overhead 
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for  the  execution  of  the  given  workload. 

The  following  points  are  relevant  to  the  problem: 

1.  In  general,  control  of  utilization  need  not  be  applied  to  ail  the  processors  in 
the  system.  Utilization  need  not  be  controlled  over  processors  with  inherently 
low  utilization.  As  well,  utilization  need  not  be  controlled  over  processors  for 
which  the  demand  per  job  is  low.  Consider,  for  instance,  a  large  data-base  sys¬ 
tem  whose  main  hardware  components  are  a  CPU  and  a  number  of  input/output 
(I/O)  devices.  The  workload  consists  of  jobs  requiring  a  very  short  service  at  the 
CPU,  followed  by  long  processing  times  at  the  T/0  devices.  An  I/O  device  can 
easily  become  congested  by  a  small  number  of  jobs  competing  for  its  service. 
However,  only  a  large  number  of  jobs  will  cause  comparable  congestion  at  the 
CPU.  A  typical  example  is  the  "United  Airlines’’  reservation  system  driven  by  an 
Airline  Control  Program  operating  s}rstem.  The  average  load  on  the  system  con¬ 
sists  of  270  transactions  per  second,  each  requiring  3.5  milliseconds  of  CPU  and 
300  milliseconds  of  I/O  (Buzen  1980).  Tor  this  data-base  system,  only  the  utili¬ 
zation  of  I/O  devices  needs  to  be  controlled  in  order  to  increase  productivity. 

2.  In  some  applications,  increasing  the  utilization  of  one  processor  is  not.  as  valu¬ 
able  as  providing  the  same  increase  to  another  processor. 

3.  In  many  cases  it  is  desirable  to  maintain  utilizations  at  levels  which  are  less 
than  the  maximum  possible,  or  to  keep  different  processors  utilized  to  different 
degrees.  Streeter  (1972),  for  instance,  suggested  a  scheme  in  which  the  value  of 
response  time  to  the  users  is  a  factor  determining  the  levels  at  which  processors 
should  be  utilized. 

In  the  productivity  algorithms  developed  in  this  thesis,  a  subset  of  the  proces¬ 
sors  can  be  handled,  weighting  of  utilizations  can  be  included,  and  various  levels 
of  desired  utilization  can  be  treated. 

We  are  interested  in  finding  an  algorithm  which  makes  the  best  selections 
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for  the  service  set  to  affect  the  desired  change  in  utilizations.  Once  such  an 
algorithm  is  found,  the  details  of  determining  optimal  utilization  levels  and  of 
applying  processor  weights  would  depend  on  the  particular  application.  In  order 
to  simplify  the  presentation  of  the  proposed  schedulers,  all  processors  are  con¬ 
sidered  to  be  equally  important.  For  the  same  reason,  the  short-term  objective 
is  maximum  utilization  for  all  processors. 

Chapter  Two  contains  a  survey  of  previous  work  in  the  area  of  macro¬ 
scheduling.  User-oriented  algorithms  are  discussed  followed  by  formal  definition 
of  system  balance  -  the  short-term  objective: of  productivity  scheduling.  There 
are  no  available  studies. of  the  problem  as  defined  here;  There  are  proposals  in 
which  an  attempt  is  made  to  combine  the  user  objectives  with  the  objective  of 
increased  productivity.  Such  proposals  are  described  with  an  emphasis  on  their 
utilization  control.  A  preliminary  approach  to  the  problem  is  the  subject  of 
Chapter  Three.  Several  new  heuristic  algorithms  are  suggested  and  results  of  a 
comparative  simulation  study  are  outlined.  Simulations  show  that  none  of  the 
macro-schedulers  examined  is  consistently  better  than  the  others.  In  Chapter 
Four,  a  more  restricted  version  of  the  problem  is  tackled.  For  a  separable, 
queueing  network  model  of  the  system  with  load  independent  service,  an  optimal 
solution  is  developed  using  mean  value  analysis.  Besides  the  optimal  algorithm, 
an  approximation  for  the  case  of  a  large  service  set  is  derived.  Chapter  Five  con¬ 
tains  analytical  studies  and  simulation  studies  of  the  macro-schedulers.  Both 
shorl-Lerm  and  long-term  effects  of  the  schedulers  are  examined.  The  results 
indicate  that  the  optimal  solution  algorithm  performs,  better  than  the  other 
schedulers.  Extensions  of  the  model  to  include  a  case  of  nonseparable  networks 
and  to  handle  deletions  from  the  service  set  are  covered  in  Chapter  Six. 
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CHAPTER  TWO 


Previous  Work 

Research  work  in  the  area  of  macro-scheduling  has  been  mostly  user- 
oriented,  due  to  the  fact  that  providing  users  with  good  service  is  the  main  pur¬ 
pose  of  a  computer  system.  In  contrast  to  the  numerous  papers  published  on 
schedulers  for  advancing  users’  objectives,  very  few  studies  consider  the  objec¬ 
tive  of  high  productivity.  No  papers  provide  a  definitive  solution  to  the  problem 
of  macro-scheduling  for  high  productivity  in  systems  with  a  variety  of  proces¬ 
sors.  In  some  studies  the  problem  is  implicitly  treated  as  part  of  an  attempt  to 
combine  the  user  objectives  with  the  objective  of  high  productivity.  In  this  sur¬ 
vey  of  previous  work,  user-oriented  schedulers  are  described  first.  After  defining 
a  short-term  objective  and  its  performance  measure,  macro-schedulers  which 
combine  user  objectives  with  the  objective  of  high  productivity  will.be  discussed 
with  an  emphasis  on  the  latter. 

2.1.  User-Oriented  Macro-Scheduling 

Greenberger  (1966)  specifies  four  user-oriented  scheduling  objectives: 

1.  reduce  average  response  time 

2.  serve  in  fair  order 

3.  limit  length  of  wait 

4.  acknowledge  customer  importance  and  urgency  of  request. 

A  fifth  objective  could  be  to  incur  little  overhead.  Regarding  the  first  objective, 
average  response  time  depends  on  service  time  distributions.  The  second  objec- 
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tive,  that  of  fair  order  of  service,  is  hard  to  define,  because  fairness  is  subjec¬ 
tive.  For  some,  service  according  to  time  of  arrival  at  the  system  seems  fair, 
while  for  others,  service  according  to  length  of  job  seems  fairer.  Other  criteria 
of  fairness  are  prevalent,  therefore,  the  question  of  fairness  will  be  ignored  in 
the  following  discussion.  A  distinction  among  algorithms  will  be  made  according 
to  whether  a  priori  knowledge  of  service  times  is  assumed.  Finally,  an  important 
feature  of  schedulers  is  their  flexibility.  Rul6s  which  involve  parameters  allow 
for  tuning  of  the  algorithm  to  suit  a  particular  situation. 

To  begin  with,  schedulers  which  do  not  acknowledge  job  urgency  are 
described.  i?irst-come-/irst-served  (FCFS)  scheduling,  under  which  jobs  are  pro¬ 
cessed  strictly  in  the  order  of  their  arrival  to  the  system,  is  probably  the  sim¬ 
plest  and  the  most  common  scheduling  rule.  This  algorithm  incurs  very  low 
overhead,  requires  no  advanced  knowledge  of  service  demands,  and  often  serves 
as  a  basis  of  comparison  for  other  schedulers.  Under  FCFS  it  may  happen,  how¬ 
ever,  that  short  jobs  wait  for  a  long  time  due  to  the  earlier  arrival  of  a  long  job, 
resulting  in  poor  average  response  time.  The  shorLesl-yob-/irsL  (S-JF)  discipline 
avoids  this  situation  by  selecting  for  the  service  set  the  job  with,  the  shortest 
processing  time.  The  result  is  fast  response  for  short  jobs  at  the  expense  of  slow 
response  for  long  jobs.  Average  response  time  improves  over  FCFS,  but  the  algo¬ 
rithm  is  strongly  biased  -  when  the  load  on  the  system  is  greater  than  system 
capacity,  long  jobs  may  wait  a  long  time.  SJF,  as  well  as  the  following  two  discip¬ 
lines,  is  based  on  the  assumption  that  service  times  are  known  exactly  in 
advance.  Later  in  this  section,  algorithms  based  on  assumptions  of  partial 
knowledge  of  service  time  will  be  discussed. 

Brinch-Hansen  (1971)  proposes  a  Aighest-response-raLio-rtexl  (KRN)  rule 
which  chooses  for  the  service  set  the  job  with  the  highest  ratio  of  the  time  it  has 
spent  in  the  system  to  its  service  time.  HRN  strikes  a  balance  between  the  FCFS 
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and  the  SJF  disciplines;  it  gives  preferential  treatment  to  short  jobs,  but  does 
not  treat  long  jobs  as  badly  as  SJF  does. 

FCFS,  SJF,  and  KRN  are  all  nonpreemptive  algorithms,  which  means  that 
once  a  job  has  been  selected  for  processing,  it  remains  in  the  service  set  until  it 
terminates.  In  preemptive  disciplines,  job  execution  can  be  interrupted  prior  to 
completion  and  the  job  removed  from  the  service  set.  The  status  of  the  job  is 
recorded  so  that  processing  can  resume  from  the  point  of  interruption.  Operat¬ 
ing  systems  which  employ  preemptive  schedulers  generate  extra  overhead  with 
every  preemption,  but  they  have  the  potential  to  provide  better  service,  e.g. 
decrease  average  response  Ume. 

Both  SJF  and  KRN  have  their  equivalent  preemptive  strategies.  Shortest- 
remaining-processmg-fime  (SRPT)  rule  is  the  preemptive  counterpart  of  SJF. 
According  to  SRPT  a  job  in  the  service  set  is  preempted  by  a  new  job  which  has 
just  arrived,  if  the  remaining  demand  of  the  active  job  is  larger  than  the  demand 
of  the  arriving  job.  This  policy  results  in  even  greater  discrimination  between 
short  and  long  jobs  compared  with  SJF.  Schrage  (1968)  proves  that  SRPT  pro¬ 
duces  the  shortest  average  response  time  possible. 

Ruschitzka  (1979)  proposes  a  common-ser vice-ratio  (CSR)  algorithm  which 
guarantees  that  among  ail  the  jobs  In  the  system,  the  jobs  in  the  service  set 
have  the  highest  ratio  of  time  spent  in  the  system  to  attained  service.  Like  KRN, 
the  principle  governing  CSR  is  equitable  sharing ,  which  is  to  provide  all  the  jobs 
in  the  system  with  the  same  virtual  processing  rate.  In  other  words,  the  princi¬ 
ple  of  equitable  sharing  means  that  when  i  jobs  require  the  use  of  a  processor 
during  t  time  units,  each  job  receives  t/i  units  of  service  time.  CSR  differs  from 
the  schedulers  mentioned  so  far,  except  for  FCFS,  in  that  it  does  not  require  a 
priori  knowledge  of  service  demands.  Three  other  preemptive  disciplines  which 
do  not  require  advanced  knowledge  of  service  times  are  discussed  next. 
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According  to  the  Zast-come-/irst~served-preemptive-resume  (LCFSPR)  rule, 
an  arriving  job  immediately  joins  the  service  set  preempting,  if  necessary,  the 
oldest,  active  job.  When  an  inactive  job  is  to  be  activated,  the  most  recent  one  to 
arrive  at  the  inactive  set  is  chosen.  Under  the  LCFSPR  rule,  preferential  treat¬ 
ment  is  provided  for  short  jobs  by  giving  them  a  chance  to  receive  their  entire 
service  before  a  long  job  arrives.  With  Poisson  arrivals  and  general  distribution 
of  service  times,  LCFSPR  produces  an  average  response  time  which  is  identical 
to  the  average  response  time  produced  fay  the  processor-sharing  rule  to  be  dis¬ 
cussed  shortly  (Kleinrock  1976). 

Under  the  round-robin  (RR)  discipline,  a  job  joins  the  service  set  for  a  fixed 
amount  of  service.  If  it  does  not  finish  within  its  allocated  quantum,  it  is 
preempted  and  replaced  by  a  job  from  the  inactive  set.  Replacement  from  the 
inactive  set  is  done  on  an  FCFS  basis.  The  quantum  length  determines  the 
characteristics  of  the  RR  scheduler.  At  one  extreme,  with  a  quantum  size 
approaching  zero,  this  rule,  called  processor-sharing  (PS),  achieves  the  equit¬ 
able  sharing  principle.  At  the  other  extreme,  when  the  quantum  size  is  infinitely 
large,  the  result  is  FCFS  scheduling.  The  amount  of  overhead  associated  with  RR 
is  a  decreasing  function  of  the  quantum  size,  as  small  quanta  lead  to  frequent 
preemptions. 

Multi-level /eedback  (FBn)  is  another  preemptive  algorithm  which  does  not 
require  a  priori  knowledge  of  service  demands  (Coffman  and  Kleinrock  1968). 
With  this  scheme  n  queues  are  provided  for  the  inactive  set.  Arriving  jobs  join 
queue  number  one.  Jobs  at  the  /th  queue  are  not  served  until  the  j—  1  lower 
queues  are  empty.  When  serviced,  a  job  receives  a  quantum  that  may  vary  from 
one  level  to  the  next.  If  a  job  at  queue  j  is  not  completed  within  the  allocated 
quantum,  it  is  then  placed  at  the  end  of  the  j+1  queue.  Jobs  at  queue  n  requir¬ 
ing  additional  service  are  returned  to  the  end  of  queue  n. 
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A  plethora  of  papers  concerning  the  above  scheduling  rules  has  been  pub¬ 
lished  containing  variations,  analyses,  and  comparative  studies.  The  most  pre¬ 
valent  approach  for  analyzing  these  algorithms  is  stochastic.  By  treating  single 
server  queues,  assuming  certain  distributions  for  the  arrival  process  and  the 
service  demand,  important  performance  measures  such:  as  mean  response  time 
are  derived.  For  more  detailed  surveys  see  Conway,  Maxwell  and  Miller  (1987), 
McKinney  (1969),  and  Kleinrock  (1976). 

A  number  of  interesting  studies  suggest  parametrized  schedulers.  Klein¬ 
rock  (1970)  proposes  the  following  preemptive  scheme.  Upon  arrival  a  job  is 
assigned  zero  priority.  While  waiting  in  the  inactive  set  priority  increases  at  rate 
a,  and  while  being  in  the  service  set  priority  increases  at  rate  /?.  The  service  set 
is  made  up  of  the  highest  priority  jobs.  By  adjusting  the  values  of  a  and  a  con¬ 
tinuum  of  seven  rules  including  FCFS,  RR,  and  LCFS  is  realized.  A  s elf-re gulatmg 
scheduler  based  on  Kleinrock’s  proposal  is  investigated  by  Bunt  and  Hume  (1972 
and  1974). 

Chua  and  Bernstein  (1974)  study  a  feedback  scheduler  in  which  the  feed¬ 
back  process  is  determined  by  functions  of  attained  service.  In  this  system,  all 
inactive  jobs  wait  in  a  single  queue.  The  job  at  the  head  of  the  queue  receives  a 
quantum  of  service.  If  it  does  not  complete  execution  within  the  allocated  time, 
it  is  placed  back  in  the  queue  at  a  position  which  is  a  funelioxi  of  its  attained  ser¬ 
vice.  By  tuning  this  function  a  variety  of  RR-like  disciplines  is  produced. 

Mitrani  and  Kine  (1977)  classify  families  of  schedulers  according  to  the 
range  of  mean  response,  times  they  can  achieve  by  adjusting  parameters  in  the 
algorithm.  Thej?-  contrive  a  theoretical  family  of  mixed  strategies  and  prove  that 
it  can  attain  a  complete  range  of  mean  response  times.  The  complete  range  of 
mean  response  times  is  the  set  of  all  possible  mean  response  times  which  can  be 
reached  with  any  scheduling  rule.  They  further  show  that  two  parametrized  fam- 
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ilies  of  strategies,  variations  of  PS,  are  near-complete.  A  family  of  strategies  is 
near-complete  if  any  element  from  the  set  of  all  possible  mean  response  times 
can  either  be  reached  by  a  strategy  from  the  family,  or  can  be  approximated  as 
closely  as  desired.  The  first  near-complete  family  is  a  processor  sharing  discip¬ 
line  in  which  different  jobs  are  allocated  different  fractions  of  processing  capa¬ 
city.  In  the  second  near-complete  family  of  strategies,  inactive  jobs  are  divided 
among  several  queues  which  are  associated  with  different  quantum  sizes.  The 
jobs  at  the  head  of  each  queue  share  the  processor.  Similar  approaches  are 
taken  by  Michel  and  Coffman  (1974)  to  determine  quantum  lengths  for  each 
queue  associated  with  the  Fffrt  scheduler,  and  by  Coffman  and  Mitrani  (1975)  to 
select  a  scheduling  discipline  in  order  to  meet  prespecified  response  time 
demands  within  a  restrictive  model. 

In  the  study  by  Mitrani  and  Hine,  response  times  are  considered  for  groups 
of  jobs  distinguished  by  external  priorities.  Providing  users  with  a  mechanism 
for  assigning  external  priorities  to  jobs,  indicating  relative  urgency,  is  a 
significant  user-oriented  objective:  In  the  simplest  external  priority  system,  a 
separate  queue  is  associated  with  each  priority  group,  and  all  jobs  with  high 
priorities  are  served  before  any  job  with  a  lower  priority.  Within  one  priority 
group,  FCFS  discipline  is  maintained.  In  order  to  avoid  a  situation  in  which  all 
users  assign  top  priority  to  their  jobs,  prices1  for  joining  priority  groups  can  be 
levied  (Ghanem  1975).  Both  preemptive  and  nonp  re  eruptive  versions  of  external 
priority  scheduling  are  common. 

Bard  (1977)  describes  a  /air  share  (FS)  scheduler  which  is  employed  by  a 
version  of  the  IBM  VM/370  interactive,  multiprogrammed  computer  system.  The 
scheduler  is  designed  to  guarantee  equal  amounts  of  CPU  time  to  all  CPU  bound 
users.  It  computes  a  CPU  consumption  rate  for  each  candidate  and  schedules 
the  next  interactive  job  so  as  to  maintain  equal  consumption  rates  for  all  jobs. 
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Jobs  are  divided  into  classes.  The  CPU  consumption  rate  of  a  class  i  job  is  the 
ratio  of  average  CPU  time  required  by  the  job  to  the  sum  of  average  user  think 
time,  average  time  spent  in  the  inactive  set,  and  average  time  spent  in  the  ser¬ 
vice  set,  where  averages  are  computed  over  class  i  jobs.  The  input  for  the 
macro-scheduler  is  provided  by  continuous  monitoring  of  jobs.  External  priori¬ 
ties  are  treated  by  multiplying  the  fixed  CPU  consumption  rate  by  coefficients 
indicating  classes’  priority. 

A  novel  concept  in  user-oriented  scheduling  is  the  poliejr-d riven  scheduler 
suggested  by  Bernstein  and  Sharp  (1971).  In  their  proposal,  a  policy  function 
describes  the  amount  of  service  which  the  system  will  try  to  offer  a  job  as  a 
function  of  its  age  -  the  time  it  has  spent  in  the  system.  Given  r,  the  amount  of 
service  the  job  has  received  so  far,  the  policy  function  specifies  the  age,  Tp,  at 
which  the  job  should  receive  r  units  of  service.  If  Tc  is  the  current  age  of  the  job, 
then  its  priority  is  the  difference  Tc-Tp.  The  fact  that  the  priority  of  a  job  is  a 
function  of  its  age  and  the  amount  of  service  it  received  allows  for  a  flexible 
specification  of  the  required  service,  and  a  good  control  of  the  jobs’  progress 
through  the  system.  The  idea  of  the  policy  function  is  generalized  by  Ruschitzka 
and  Fabry  (1977)  and  further  analyzed  by  Ruschitzka  (1978  and  1980).  Policy 
functions  have  been  incorporated  into  production  systems,  such  as  the  system 
resources  manager  (SRM)  of  IBM  0S/VS2  MVS  operating  system  (IBM  1978),  as 
well  as  research-oriented  sj^stems  like  KYDRA  (Levin  et.  al.  1975). 

In  some  studies  of  user-oriented  scheduling,  partial  knowledge  of  service 
times,  loss  functions,  and  deadlines  are  considered.  In  general,  compared  with 
partial  knowledge  of  service  times,  precise  knowledge  permits  lower  average 
response  time.  Studies  involving  loss  functions  concern  algorithms  which 
schedule  jobs  so  as  to  minimize  the  total  loss  incurred  by  delayed  completion 
times.  A  set  of  functions  specifying  loss  due  to  delayed  completion  is  assumed. 
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Among  nonpreemptive  schedulers  with  linear  loss  functions,  the  rule  SEJF/C,  Le. 
shortest  expected  job  to  loss  rate  ratio,  is  optimal  (Rothkopf  1986).  When 
preemption  is  allowed  and  the  distributions  of  service  times  are  known,  a  smal¬ 
lest  rank  (SR)  scheduler  is  optimal  (Sevcik  1974).  The  SR  scheduler  ranks  jobs 
according  to  the  ratio  of  expected  investment  to  expected  payoff.  The  job  with 
the  smallest  rank  is  selected  for  the  service  set.  Deadline  scheduling  is  con¬ 
cerned  with  providing  a  schedule  which  guarantees  job  completion  within  a 
specified  time  limit  (Serlin  1972,  and  Kim  and  Naghibzadch  1980). 
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Table  i.  A  general  characterization  of  user-oriented  schedulers 


A  summary  of  the  characteristics  of  various  user-oriented  schedulers  is 
given  in  Table  1.  Table  1  is  a  generalization  intended  to  provide  a  feel  for  the 
various  scheduling  disciplines.  There  are  exceptions  to  the  characteristics  men¬ 
tioned  in  the  table,  mainly  due  to  the  dependency  of  the  characteristics  on  the 
distribution  of  service  times. 


2.2.  The  Short-Term  Objective  and  Its  Performance  Measure 

Early  studies  of  productivity  algorithms  attempted  to  optimize  the  use  of 
just  one  computer  resource.  Generally,  it  is  the  CPU  (Wallace  and  Mason  1969, 
Bunt  1974);  sometimes  it  is  mam  storage  utilization  (Belady  and  Kuehner  1969), 
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on  of  the  channel  between  two  levels  of  storage  (Reiter  1967).  In 
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his  Ph.D.  thesis,  Denning  (1968a)  investigated  the  issue  of  dynamically  allocating 
limited  processor  and  main  storage  resources  among  users.  Kis  solution  was 
system  balance,  a  situation  whereby  the  demand  of  the  service  set  matches  the 
available  equipment.  Note  that  balancing  in  this  context  does  not  refer  to  equal¬ 
izing  the  load  among  resources,  but  to  bringing  the  load  on  a  resource  to  a 
specified  leveL 

Denning’s  thesis  deals  specifically  with  paging  systems  in  which  the  equip¬ 
ment  consists  of  main  storage  and  a  group  of  identical  processors.  There  are 
two  types  of  demand  in  such  a  system.  Storage  demand  is  defined  as  the  ratio  of 
the  number  of  pages  required  by  the  working  sets  of  all  active  jobs  to  the 
number  of  pages  in  main  storage.  Processor  demand  is  defined,  for  a  multipro¬ 
cessing  system*,  as  the  ratio  of  the  number  of  processors  immediately  needed 
by  all  active  jobs  to  the  number  of  processors  in  the  system.  Corresponding  to 
system  demand,  system  balance  is  represented  as  a  pair  (a,  /S),  where  a  is  pro¬ 
cessor  balance  and  jS  is  storage  balance.  Processor  balance  and  storage  balance 
are  target  values  for  the  macro-scheduler.  The  objective  of  the  macro- 
scheduler  in  such  a  scheme  is  to  achieve  system  balance,  i.e.  to  make  processor 
demand  equal  to  a,  and  storage  demand  equal  to  /?.  The  values  of  c*  and  §  are  to 
be  determined  by  the  system  administrators. .  Denning  proposed  making  a  and  @ 
less  than  1.0  for  the  following  reasons: 

o 

1.  the  utilization  of  processors  should  be  kept  at  a  leveL  which  ensures  accept¬ 
able  response  time  to  all  users,  and 

2.  the  system  should  maintain  a  reservoir  of  pages  in  main  storage  to  avoid 
thrashing. 

Denning  outlined  an  algorithm  which  is  nearly  optimal  for  reaching  storage 
balance.  The  algorithm  solves  a  Knapsack  problem,  i.e.  it  loads  main  storage  to 


*  A  different  definition  is  used  in  the  case  of  single-processing  systems. 
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the  desired  level  according  to  the  principle  of  maximum  value  per  unit  of 
■weight;  given  an  objective  function  which  determines  the  value  of  loading  a  job, 
the  jobs  with  the  highest  ratio  of  value  to  storage  demand  are  selected.  The 
problem  of  how  to  balance  both  storage  and  processors  was  left  unsolved. 

Denning  did  not  attempt  to  solve  the  problem  of  macro-scheduling  for  a 
system  in  which  jobs  require  service  at  several  processors,  but  his  objective  of 
system  balance  is  still  applicable.  In  the  context  of  a  system  with  a  variety  of 
processors,  system  balance  is  defined  as  a  situation  in  which  the  processing 
demand  of  the  service  set  is  at  some  prespecified  levels.  A  scheduler  designed 
to  reach  system  balance  for  the  next  scheduling  interval  is  a  balancing 
scheduler. 

Before  discussing  system  balance  in  greater  detail,  it  is  important  to  stress 
that  system  balance  is  a  short-term  objective.  We  assume  that  by  repeatedly 
trying  to  achieve  system  balance  at  every  scheduling  interval,  the  elapsed-time 
needed  to  execute  a  given  workload  could  be  shortened  compared  with  the 
elapsed-time  needed  to  execute  the  same  workload  under  a  neutral  scheduler. 
This  assumption  will  be  examined  in  Section  5.3. 

Processing  demand  of  the  service  set  is  a  weighted  sum  of  processor 
demands.  Processor  demand  is  defined  as  the:  utilization  of  the  processor  during 
a  scheduling  interval.  The  demand  on  processor  j  will  be  denoted  by  Uj(S), 
specifying  dependence  on  the  make-up  of  the  service  set,  Processor  balance 
which  is  the  target  utilization  value,  set  so  as  to  achieve  high  productivity,  will 
be  denoted  by  o ij.  Processor  imbalance  is  the  absolute  value  of  the  difference 
between  processor  balance  and  processor  demand,  |  <Xj-Uj(S )  j .  The  short-term 
objective  of  a  productivity  scheduler  is  to  minimize  system  imbalance,  the  aver¬ 
age  of  all  processor  imbalances,  —  J]  {  a.j-Uj(S)  j ,  where  if  is  the  set  of  proccs- 

*  K 

sors  over  which  utilization  is  controlled,  J:  —:\K\.  The  reason  for  including  an 
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average  in  the  definition  of  system  imbalance  is  to  keep  it  independent  of  the 
number  of  processors. 

As  mentioned  in  Chapter  One,  our  short-term  goal  is  to  find  the  optimal 
algorithm  for  reaching  system  balance,  regardless  of  the  relative  importance  of 
processors,  or  the  values  of  processor  balance.  In  order  to  simplify  the  presen¬ 
tation  of  the  proposed  schedulers,  all  processors  are  considered  equally  impor¬ 
tant,  in  the  sense  that  decreasing  the  imbalance  of  one  processor  is  as  impor¬ 
tant  as  decreasing  the  imbalance  of  another  processor  by  the  same  amount.  If 
in  some  application  the  importance  of  processors  varies,  weights  can  be 
appended  to  the  expressions  for  processor  imbalance.  As  well,  for  the  sake  of 
simplicity,  the  values  of  processor  balance  are  set  to  1.0, 
ai  =  ag  =  =  at  =  1.0,  Le.  full  utilization.  Where  appropriate,  we  shall  indi¬ 

cate  how  to  employ  the  suggested  algorithms  for  values  of  processor  balance 
other  than  1.0. 

With  the  above  simplifications,  the  values  of  processor  imbalance  become 

1 

nonnegative,  making  the  objective  of  minimizing  —  j  l—Uj(S)\,  equivalent  to 

the  objective  of  maximizing  —  Uj(S).  The  short-term  objective  for  increasing 

k  je  K 

productivity  is,  therefore,  to  maximize  the  average  of  processor  utilizations.  We 
shall  refer  to  the  average  of  processor  utilizations  as  system  demand,  denoted 
by  P  (5 ): 

^(S)  =  7  £  C'V(S) 

In  performance  measurement  and  evaluation,  workload  is  typically  character¬ 
ized  in  terms  of  the  number  of  jobs  which  belong  to  a  particular  class,  where 
classes  are  distinguished  according  to  demands  placed  on  resources.  The  follow¬ 
ing  notation  will  be  used: 

M  =  set  of  job  classes 
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Xi  ~  throughput  of  class  i,  i  £  M 

iy  =  expected  total  processing  demand  of  a  class  i  job  on  processor  j 

D%  ~  2  ~  expected  total  processing; demands  of  a  class  i  job 

K 

on  all  processors 

Consider  the  average  of  processor  utilizations  over  a  time  interval  which  is 

longer  than  the  length  of  a  scheduling  interval,  P  -  —  2  Uj.  The  notation  for  P 

k  Jfe  K 

and  for  U $  reflect  the  fact  that  they  do  not  depend  on  the  make-up  of  a  particu¬ 
lar  service  set.  By  definition  Uj  -  2  hence: 

ieM 


p  =  j  E  u,  =  j  2  2  a,  t*  =  i  s  ,v  2  tit  =  £  S  ** Di 

K  JeK  K  jeK  ieM  fC  ie  M  je  K  K  ie  M 

This  proves  that  P,  and  therefore  P(S),  equal  a  weighted  sum  of  class 


throughputs.  The  weighting  of  the  throughput  of  class  i  is  proportional  to  the 
expected  total  processing  demand  of  a  class  i  job.  Consequently,  in  order  to 
maximize  P(S),  increasing  the  throughput  of  a  class  with  large  processing 
demands  is  more  important  than  providing  the  same  increase  to  a  class  with  low 
demands.  By  measuring  P(S)  with  throughputs,  it  is  possible  to  assign  weights 
to  different  classes.  However,  in  the  following  chapters,  P(i5)  will  be  measured 
with  utilizations  in  order  to  keep  it  independent  of  the  division  of  workload  into 
classes.  Measuring  P(S)  with  utilizations  is  equivalent  to  calculating  P(S)  as  a 
weighted  sum  of  throughputs,  -where  every  job; is  in  a  class  of  its  own. 


The  measure  P  is  not  new.  Leroudier  and  Potier  (1976)  and  Badei  and 
Leroudier  (1978)  use  the  sum  of  utilizations  as  a  long-term  performance  meas¬ 
ure  and  call  it  "dilatation".  They  prove  that  apart  from  indicating  utilization  of 
processors,  dilatation  is  a  measure  of  the  ratio  of  real  execution  time  of  $  jobs 
with  no  multiprogramming  to  real  execution  time  of  s  jobs  with  a  multiprogram¬ 
ming  degree  s.  In  other  words,  dilatation  measures  the  efficiency  of  multipro¬ 
gramming,  or  the  degree  of  parallelism  in  the  use  of  processors.  Leroudier  and 
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Potier  show  that  in  a  system  where  a  virtual  memory  is  employed,  having  the 
CPU  and  an  I/O  processor  utilized  to  the  same  degree  optimizes  the  measure  of 
dilatation,  and  Bade!  et.al.  (1975)  prove  that  increasing  dilatation  improves  aver¬ 
age  response  time  in  the  system. 

Before  investigating  macro-schedulers  for  systems  with  a  variety  of  proces¬ 
sors,  we  note  some  other  applications  of  the  balance  objective.  Denning  et.al. 
(1976,  see  also  Brmch-Hansen  1973)  in  a  study  of  virtual  memory  systems  use 
the  balance  of  the  paging  device  as  a  criterion  for  macro-scheduling.  Geek 
(1979)  reports  on  balancing  load  over  CPU,  I/O  and  main  storage  in  a  B6700  with 
a  virtual  memory  operating  system  The  balancing  is  not  done  by  controlling 
the  mix  in  the  service  set,  but  by  adjusting  parameters  which  partially  shift  load 
among  resources.  In  systems  where  a  virtual  memory  is  employed,  load  can  be 
shifted  between  main  storage  and  CPU-I/0  by  controlling  the  frequency  with 
which  the  usefulness  of  main  storage  pages  is  tested,  or  by  controlling  the  time 
limits  defining  the  working  set  of  a  job.  In  the  case  of  the  B6700  system,  a  look¬ 
ahead  paging  out  is  performed  by  a  special  task  every  three  seconds.  The  task 
randomly  chooses  pages  to  be  overlayed  among  all  pages  which  have  not  been 
referenced  within  a  given  time  interval.  Load  control  is  achieved  by  tuning  the 
parameter  which  defines  the  percentage  of  main  storage  pages  to  be  overlayed 
in  advance  within  one  minute.  According  to  Geek,  an  improvement  of  system 
throughput  between  8%  and  18%  has  been  gained  with  this  method. 

Finally,  with  regard  to  the  selection  of  processor  balance  points,  Leroudier 
and  Potier  (1976)  show  that  levelling  the  utilization  of  processors,  i.e.  having  all 
processors,  except  for  the  paging  device,  utilized  to  the  same  degree  tends  to 
increase  throughput.  These  results  corroborate  earlier  findings  by  Denning 
(1968b,  1969,  and  1971),  Wulf  (1969),  and  Buzen  (1971).  In  real  systems,  how¬ 
ever,  levelling  utilization  over  all  processors  is  often  avoided,  e.g.  CPU  utilization 
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would  be  0.85  and  channel  utilization  would  be  0.4. 

2.3.  A  Survey  of  Balancing  Strategies 

Four  macro-schedulers  in  which  the  objective  of  high  productivity  is  pur¬ 
sued  are  outlined.  In  all  of  them,  systems  with  a  variety  of  processors  axe  con¬ 
sidered,  such  that  each  job  has  demands  for  main  storage  and  for  a  few  specific 
processors.  In  the  first  two  studies  it  is  assumed  that,  for  every  job,  the 
scheduler  has  information  about  demands  for  storage  and  for  processing  per 
unit  of  virtual  service  time.  Virtual  service  time  of  a  job  is  the  total  processing 
time  received  by  the  job  during  a  measurement  interval. 

Mahl  (1970)  in  his  PhuD.  thesis  tried  to  find  a  set,  S',  of  active  jobs  which 
maximizes  the  economic  function; 


E  —  CtWi 
l  6  5 

under  the  constraints  of: 

I.  main  storage  capacity  Y1  bx5t  £  B  and 

i€  S 


II.  processors  capacity 


u>i  +  Oi'j  u>r  <  1  for  ail  i€  5 

/€  K 

where: 

Ci  =  external  priority  of  job  i 

Wi  =  effective  progress  rate  of  job  i,  I.e. 

_  time  job  i  is  processed  on  any  -processor 

scheduling  interval 

bi  s  amount  of  main  storage  required  by  job  i 


if  Wi  =  0 
if  Wi  >  0 


B  =  main  storage  capacity 
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K  =  set  of  processors 

Ojj  =  processing  demand  of  job  i  on  processor  j  per  unit  of 
virtual  service  time  of  job  i 

Pq  =  an  integer  representing  the  preemptive,  micro-scheduling 
priority  of  job  i  for  processor  j 
Pij  =  Pi’j  <==>  i  -  i' 

Pii  <  Pi  •  j  <  ==>  job  i  has  higher  priority  than  i'  for  processor  j 

In  this  approach,  the  macro-scheduLer  is  given  a bi,  and  C*  to  determine 
and  u>i  for  all  jobs  and  processors.  In  order  to  avoid  an  exhaustive  examina¬ 
tion  of  all  possible  selections  and  micro-scheduling  priority  assignments  for  the 
service  set,  the  following  procedure  is  suggested  by  Mahl.  Each  processor  j  is 
assigned  an  initial  weight,  Hj  ( Hm  denotes  the  weight  assigned  to  main  storage), 
and  every  job  i  an  initial  progress  rate,  w Then  the  desirability  of  a  job, 

dx  =  Ci/'(biHm+  OijWiHj),  is  computed.  New  values  for  Wi  are  selected 
jzK 

according  to  the  principle  that  jobs  having  a  larger  should  have  a  larger  wt. 

Also,  new  values  for  Hj  are  chosen  such  that  a  processor  whose  utilization, 

Y.  a. ijWir  is  much  smaller  than  1.0  has  its  weight  decreased.  Having  chosen  the 
ie  5 

jobs  for  the  service  set,  the  u^’s  are  further  approximated  by  solving  the  equa¬ 
tions: 

Wi  +■  ^  a ^  =  1  for  all  i€  S 

where  <  p Vf  <==> 

To  explore  the  productivity  control  aspect  of  such  a  scheduler,  we  can 

inspect  its  behaviour  assuming  that  all  jobs  have  the  same  external  priority,  i.e. 

Ci  =  C  for  ail  i.  Under  this  assumption,  the  objective  function  becomes 

E  -  C'ZvJi,  i.e.  the  scheduler  will  try  to  maximize  the  sum  of  the  effective  pro- 
te  S 
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gress  rates  for  the  jobs  in  the  service  set.  The  progress  rate  of  a  job,  according 
to  the  procedure  mentioned  above,  is  proportioned  to  its  desirability.  The 
desirability,  in  turii,  is  inversely  proportional  to  the  weighted  sum  of  the  relative 
amount  of  resources  consumed  by  the  job,  V]  where  weights  are 

proportional  to  the  utilization  of  resources.  The  effective  progress  rates  are  also 

determined  by  the  micro-scheduling  priorities  which  are  increasing  with  the 

relative  demands:  Pij  <  Pi’j  <~=>  <  a^.  Overall  we  see  that  the  rational 

governing  this  scheduler  is  to  load  the  resources  to  capacity  by  determining 

progress  rates  for  the  active  jobs.  The  problem  with  this  approach  is  that  there 

is  no  guarantee  that  the  progress  rates  set  by  the  above  procedure  will  be  the 

same  as  the  actual  progress  rates  during  the  upcoming  scheduling  interval.  As  a 

result,  we  have  no  proof  that  the  utilization  of  processors,  ^  4  for  processor 

ie  S 

j,  will  be  high  because  of  the  uncertainty  in  the  values  of  tu*.  Mahl’s  approach  is 
restrictive  since  preemptive  priority  micro-seheduling  is  presumed.  Moreover, 
since  micro-scheduling  priorities  are  determined  by  the  macro-scheduler,  ser¬ 
vice  at  some  processors' could  become  poorer.  Consider;  for  example,  a  channel 
serving  jobs  in  an  order  designed  to  minimize  disk  arm  movement.  Micro¬ 
scheduling  priorities  set  by  the  macro-scheduier  would  probably  conflict  with 
the  order  of  service  necessary  to  decrease  arm  movement,  resulting  in  slow  ser¬ 
vice  at  the  channel 

The  approach  of  Kameda  and  Gotlieb  (1977,  see  also  Kameda  1975)  is  some¬ 
what  similar  to  Mahl.  They  posbilate  a  free  economy  system  in  which  each  job  is 
seen  as  a  consumer  who  tries  to  minimize  the  waiting  time  for  service  within  a 
budget  constraint.  The  strategy  of  a  consumer  is  to  choose  a  combination  of 
bids,  which  are  identical  to  priorities,  in  order  to  minimize  the  expected 
total  waiting  time,  Tit  for  a  unit  of  processing.  The  following  objective  function  is 
associated  with  job  i: 
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under  the  constraint: 


T'i  ~  X!  /;(Pi;  )ay 

j*K 


Cx~  Tj  PijCkj  +  PmbiTi  and  Pvj  ^  0  for  all  j €  K 
jeK 

Where: 

fj(p)  =  a  function  -which  gives  the  expected  waiting  time,  per  unit 
of  processing  on  processor  j,  for  a  job  with  priority  p 
Ct  =  the  value  of  a  unit  of  processing  time  for  job  i 
pm  =  the  "price"  of  a  unit  of  main  storage 

Even  though  the  bidding  is  done  separately  for  each  processor,  the  authors 
demonstrate  the  existence  of  a  coherent  priority  queueing,  i.e.  a  priority  d*. 
which  is  characteristic  of  the  job  regardless  of  processor,  where: 


<k  =  if  pif  >0 

U-Pij 

r  ^  dfj(Pij) 

*  -  Pii = 

Thus,  at  each  invocation  time,  the  jobs  with  the  highest  bids  are  selected  for  the 


service  set. 


In  a  comparative  static  analysis,  Kameda  and  Gotlieb  reveal  the  produc¬ 
tivity  control  aspect  of  their  scheduler,  which  is  similar  to  the  control  suggested 
in  Mahl’s  study.  They  prove  that  an  increase  in  the  congestion  of  one  processor 
leads  to  a  decrease  in  the  priority  of  all  jobs  serviced  by  this  processor,  pro¬ 
vided  other  conditions  remain  fixed.  Also,  jobs  with  larger  demand  for  a  cong¬ 
ested  processor  will  lose  more  priority  than  other  jobs.  This  works  towards  elim¬ 
ination  of  bottlenecks,  and  levelling  off  processor  utilizations. 

A  major  problem  with  the  study  by  Kameda  and  Gotlieb  is  the  determina¬ 
tion  of  i.e.  the  function  which  maps  the  priority  of  a  job  to  the  amount  of 

time  the  job  has  to  wait  in  order  to  receive  a  unit  of  processing  from  processor 
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j.  For  implementation  purposes  a  rough  approximation  for  dt  is  suggested,  but 
the  exact  determination  of  the  fj(p)  functions  remains  a  crucial  missing  link. 
This  approach,  like  Mahl’s,  is  restrictive  since  preemptive  priority  micro 
scheduling  is  presumed,  ■  and  since  micro-scheduling  priorities  are  determined 
by  the  macro-scheduler. 

Badel  and  Leroudier  (1978)  describe  yet  another  macro-scheduling  algo¬ 
rithm  for  balancing  processors.  Using  a  hybrid  approach,  simulation  and 
analysis,  they  study  a  central  server  model  of  a  virtual  memory  system.  Their 
scheduler  is  made  to  regulate  the  service  set  such  that  inter-request  intervals 
for  a  processor  equal  the  processor's  mean  service  time.  The  objective  is  to 
equalize  utilization  of  all  processors,  except  for  the  paging  device.  For  optimum 
system  performance,  the  utilization  of  the  paging  device  is  kept  at  about  0.5 
(see  also  Denning  et.al..  1976).  The  macro-scheduler  selects  a  job  for  loading 
from  the  inactive  set,  /,  using  the  following  formula: 


m 


Min 

ie  / 


E  9/ 


i=t 


U* 


ut 


CPU 


TFDS  .  .  I 

+  Tro--(s  +  1)\ 


where: 


m  as  number  of  I/O  processors 


9/ 


if 


TFD, 


TIO 


>  1 


if 


.  ,  TFD, 

(1  -  mjj)  •  j 


otherwise 


<7j-  is  a  weighting  factor 
s  =  multiprogramming  level 
Uj  =  utilization  of  I/O  processor  j,  j  =  1  ,..,,771 
UCPU  —  utilization  of  CPU 
TFDj  s  mean  service  time  of  processor  j 
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TIOij  =  mean  time  interval  between  consecutive  I/O  requests  by- 
job  i  to  processor  j 

7 j  =  a  coefficient  related  to  physical  characteristics  of  processor  j 

The  authors  do  not  specify  how  to  determine  jy  A  strong  reservation  con¬ 
cerns  the  applicability  of  this  macro-schcdulcr  to  more  general  cases,  "While  for 
some  processors  mean  service  time  can  be  easily  predicted,  for  many  others 
service  times  are  a  function  of  demand.  The  macro-scheduler  becomes  imprac¬ 
tical  in  instances  where  we  do  not  know  how  to  predict  the  mean  service  time  of 
a  processor.  Badel  and  Leroudier  report  success  with  this  type  of  scheduler.  In 
a  simulation  study,  system  dilatation,  which  is  the  sum  of  processor  utilizations, 
increased  from  a  ievel  of  1.3  without  mixture  control  to  levels  between  1.5  to  3.0, 
depending  on  the  load. 

The  last  balancing  strategy  considered  is  employed  by  the  swapping 
analysis  algorithm  which  is  part  of  SRM  -  the  macro-scheduler  of  IBM  MVS 
operating  system  (Lynch  and  Page  1974,  IBM  1978,  and  Buzen  1978).  In  the  MVS 
system,  jobs  are  divided  into  performance  groups  according  to  performance 
objectives  determined  by  the  system  administrator.  Typically,  batch,  time¬ 
sharing,  and  transaction  processing  correspond  to  diflerenl  performance 
groups.  A  system  may  have  several  batch  performance  groups  if  separate  per¬ 


formance  objectives  are  required  for  several  different  categories  of  batch  jobs. 
The  same  is  true  of  time-sharing  and  of  transaction  processing.  The  concept  of 
domain  is  used  to  control  the  service  provided  to  performance  groups.  A  domain 
is  a  region  in  main  storage  into  which  jobs  (working  sets)  from  one  or  more  per¬ 
formance  groups  can  be  loaded.  Not  all  jobs  which  correspond  to  a  domain  are 
active  simultaneously.  The  number,  of  jobs  that  a  domain  currently  contains  is 
the  multiprogramming  ievel  of  the  domain.;  The  service  provided  to  perfor¬ 
mance  groups  is  controlled  by  assigning  performance  groups  to  domains  and  by 
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varying  a  target  multiprogramming  level  for  each  domain. 

Within  individual  domains,  allocation  of  resources  among  jobs  which 
correspond  to  a  domain  is  controlled  by  exchange  swapping.  Exchange  swapping 
is  performed  by  the  swapping  analysis  algorithm  which  receives  three  swap 
recommendations  for  every  candidate.  These  recommendations  are  produced 
by  three  other  algorithms:  the  CPU  load  adjusting,  the  I/O  load  adjusting,  and 
the  workload  management.  The: :  swapping  analysis  algorithm  evaluates  a 
weighted  sum  of  these  recommendations,  where  the  weights  are  parameters  of 
the  system.  The  job  with  the  highest  computed  value  is  swapped.  The  workload 
management  algorithm  evaluates  each  swap  in  terms  of  its  effect  on  the  rate  at 
which  the  job  rs  supposed  to  receive  service,  i.e.  the  objective  is  user-oriented. 
The  objective  associated  with  the  other  two  algorithms  is  high  productivity.  A 
recommended  value,  RV^,  is  computed  by  each  of  these  algorithms  as  follows: 

RVij  =  D2  ry 

where: 

Dj  =  the  distance; processor  j  is  from  its  balance  point 
nj  s=  the  expected  amount  of  processing  per  second  to  be  received  by 
job  i  from  processor  j 

A  sign  is  affixed  to  the  RVy  indicating  the  direction  of  the  swap,  in  or  out  of 
main  storage.  In  the  case  of  the  I/O  load  adjusting,  a  logical  channel  is  con¬ 
sidered  a  processor.  If  more  than  one  of  the  logical  channels  needed  by  a  candi¬ 
date  is  underutilized/ overutilized,  then  Dj  and  ry  are  computed  on  the  logical 
channel  with  the  greatest  imbalance.  In  a  simulation  study  reported  by  Rosen- 
feld  (1977),  he  concluded  that  the  load  adjusting  algorithms  served  to  improve 
throughput,  especially  when  the  resources  were  not  overutilized.  In  reality, 
however,  performance  improvement  is  needed  more  at  times  of  heavy  workload 
than  at  times  of  light  workload.  Improvements  of  5%  to  20%  were  observed 
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depending  on  system  load. 

To  summarize,  in  this  chapter  we  have  surveyed  various  user-oriented 
schedulers,  and  defined  system  balance  making  it  a  short-term  objective  in 
order  to  increase  productivity.  Four  strategies  for  balancing  a  system  with  a 
variety  of  processors  were  discussed,  three  of  which  are  not  generally  practical. 
In  the  next  chapter,  new  heuristic  algorithms  for  system  balance  are  proposed. 
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CHAPTER  THREE 


Preliminary  Approach 

Due  to  the  small  number  of  productivity  macro-scheduling  algorithms  with 
general  applicability,  an  initial  approach  to  the  problem  of  macro-scheduling  for 
high  productivity  will  be  the  design  and  the  testing  of  new  schedulers.  Four  new 
heuristic  algorithms  for  increasing  productivity  will  be  presented.  A  simulation 
comparative  study  follows  this  description.  Two  major  points  relate  the  new 
schedulers  to  earlier  works  discussed  in  Section  2.3.  The  first  is  that  in  three 
algorithms  an  attempt  is  made  to  select  the  job  which  will  contribute  the  most 
to  bringing  processor  utilizations  to  the  same  leveL  The  second  point  is  that, 
like  the  SRM  productivity  scheduler,  the  suggested  algorithms  are  strictly 
designed  for  utilization  control,  such  that  their  ability  to  control  utilization  can 
be  compared. 

3.1.  Heuristic  Algorithms  for  System  Balance 

The  first  algorithm,  HCLUP,  is  based  on  a  simple  heuristic.  The  input  to 
this  macro-scheduler  consisrts  of  processor  utilizations,  and  of  expected 
amounts  of  processing  per  time  unit  to  be  delivered  to  each  candidate.  The 
HCLUP  algorithm  selects  the  candidate  which  is  predicted  to  be  the  highest  con¬ 
sumer  of  the  least  utilized  processor.  It  identifies  Umvtl  —  Mill  t7,,  and  selects  a 

ie  k  J 

job  according  to  the  formula: 

Max  Txjffa 

XG  1 


where: 
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Uj  =  utilization  of  processor  j 

K  =  set  of  processors,  k  =  \K  | 

Ty  =  expected  amount  of  processing  per  time  unit  to  be  received  by  job  i 

from  processor  j 

I  =  inactive  set 

LCMUP  operates  like  KCLUP,  requires  the  same  information,  but  deals  with 
the  most  as  opposed  to  the  least  congested  processor.  LCMUP  selects  for  the 
service  set  the  job  which  is  expected  to  be  the  lowest  consumer  of  the  most  util¬ 
ized  processor.  It  identifies  U^ttz  —  Max  Uj,  and  selects  a  candidate  to  be  loaded 

3  €  % 

according  to  the  formula: 

Min  7*  ri/riT 

ie/ 

If  the  values  of  processor  balance  are  less  than  1.0,  HCLUP  and  LCMUP 
should  be  used  as  two  components  of  one  productivity  scheduler.  Given  the 
values  ak  for  processor  balance  points,  the  productivity  scheduler  will 

determine  the  most  unbalanced  processor  according  to  Max  j  a;  — j .  If  >  Uj, 

i.e.  if  the  processor  is  underutilized,  then  HCLUP  will  be  invoked,  otherwise, 
LCMUP  will  be  invoked.  Combining  HCLUP  and  LCMUP  in  one  scheduler  elim¬ 
inates  a  disadvantage  associated  with  an  exclusive  use  of  HCLUP.  Under  HCLUP 
scheduling,  a  user  could  sharply  reduce  his  response  time  by  artificially  placing 
a  high  demand  on  an  underutilized  processor. 

BaLANC  is  a  new  algorithm  developed  to  increase  and  to  balance  utiliza¬ 
tions  of  all  processors.  The  algorithm  is  best  explained  in  terms  of  a  multi¬ 
dimensional  geometric  model  which  describes  the  space  of  processor  utiliza¬ 
tions  (Figure  l).  Each  axis  in  this  space  corresponds  to  a  processor’s  utilization. 
In  the  model,  we  identify  the  system  operating  point,  the  point  of  system  bal¬ 
ance,  and  the  target  direction.  System  operating  point  is  determined  by  the 
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utilization  of  processors  at  some  invocation  time.  Because  the  intent  of  the 
strategy  is  balancing  utilization  of-  ail  processors,  a  target  direction  is  defined 
from  the  system  operating  point  to: the  point  of  system  balance. 


SYSTEM 

OPERATING 

POINT 


Ul 


Fig.  1.  A  model  for  the  BALANC  algorithm  with 

2  processors  and  3  candidate  jobs 

A  job  in  this  scheme  is  represented  by  a  direction  determined  by  process¬ 
ing  demands  per  unit  of  virtual  service  time.;  Assuming  that  the  predictions  of 
the  demands  per  unit  of  virtual  service  time  are  correct,  the  actual  amounts  of 
processing  which  will  be  delivered  lo  an  active  job  during  the  next  scheduling 
interval  are  represented  by  a  point  along  this  direction.  The  exact  coordinates 
of  the  point  are  unknown  in  advance,  and  depend  on  the  mix  with  which  the  job 
will  reside.  The  effect  of  loading  a  new  jab  is  interpreted  as  moving  the  system 
operating  point  in  the  direction  of  this  job.  Consequently,  among  all  candidates 
BALANC  singles  out  the  one  whose  direction  is  closest  to  the  target  direction. 

A  formula  for  BALANC  is  developed  using  the  following  notation: 

U!f  =  current*  utilization  of  processor  j,  j  €  K 

Uj  =  a  coordinate  corresponding  to  processor  j,  of  a  point  which  lies  on  the 


*  Current  utilization  means  a  utilization  measured  over  an Interval  which,  euds  at  the.' current 
invocation  time.  The  length  of  the  interval  is  a  parameter  of  the  macro-scheduler. 
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line  between  the  system  operating  point  and  the  point  of  system  balance 
=  dij/a u  =  the  ratio  of  the  demand  of  job  i  on  processor  j  to  the 
demand  of  job  i  on  processor  1.  Note  that  <5tl  =  UO.  Processor  1  is  arbi¬ 
trarily  singled  out  in  the  definition  of  <5#.  Any  other  processor  from  the  set 
K  could  have  equally  been  selected. 

Notice  that  up  to  this  point,  Uj  has  been  used  to  indicate  current  utilization  of  a 
processor.  This  notation  will  be  resumed  following  the  discussion  of  the  BALANC 
algorithm.  With  current  system  operating  point  at  and  with  system 

balance  at  the  equations  of  the  target  direction  line  are: 

Ui  -  01  =  =  Uk  -  U£ 

i  -  u\  .  i  -  u§ 

and  its  direction  cosines  are: 


The  direction  cosines  of  job  i  are: 


where  Z  =  -  /  E)  (1  -  t/|)£ 
V  je  ff 


a 


i  1 


2-i 


where  Zi  =  -v  /  E 

V  j^K 


If  is  the  angle  between  the  direction  of  job  i  and  the  target  direction,  we  have: 


cos©*  =  E 

jzK  zi 


j_  6i}  (1  -  up 
Z  & 

V  j€ K 


Therefore,  selecting  a  candidate  whose  direction  is  closest  to  the  target  direc¬ 
tion  amounts  to  choosing  job  x  such  that: 


Max 

16/ 


cos©2  =  Max 

xzl 


E  sx,  (i  —  u%) 


j€  K 


This  equation  is  the  selection  criterion  of  BALANC.  The  chosen  job  has  demands 
which  are  low  from  congested  processors  and  high  from  underutilized  ones. 


Intuitively,  one  would  expect  BALANC  to  perform  better  than  HCLUP  and, 
for  the  same  reasons,  to  perform  better  than  LCMUP.  HCLUP  selects  for  loading 
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the  job  which  is  the  highest  consumer  of  the  least  utilized  processor,  but  by  this 
action  it  might  alter  utilization  of  other  processors,  creating  new  congestion 
points  without  improving  system  imbalance.  In  BALANC,  on  the  other  hand, 
demands  on  ail  processors  are  considered:  an  attempt  is  made  to  shift  the  sys¬ 
tem  operating  point  towards  system  balance,  less  congestion  and,  presumably, 
an  increase  in  the  average  of  processor  utilizations,  P(S ). 

Another  argument  in  favour  of'  BALANC  is  the  type  of  input  it  requires.  The 
use  of  HCLUP  and  LCMUP  requires  knowledge  of  demands  under  competition , 
which  are  the  amounts  of  processing  per  time  unit  to  be  received  by  a  job.  The 
use  of  BALANC  requires  knowledge  of  intrinsic  demand  ratios,  which  are  ratios 
of  job’s  demands  regardless  of  its  competition  with  other  active  jobs.  Demands 
under  competition  are  difficult  to  predict,  since  thejr  are  determined  by  the 
intrinsic  demands  and  by  the  interactions  among  active  jobs.  The  ratios  of  a 
job’s  intrinsic  demands  can  be  estimated  from  monitored  data  by  taking  the 
ratios  of  the  amounts  of  processing  received  by  the  job.  If  the  demand  pattern 
of  the  job  is  cyclic  and  repetitive,  the  estimation  of:  the  ratios  of  intrinsic 
demands  from  the  ratios  of  processing  received  will  be  accurate  and  indepen¬ 
dent  of  competition  with  other  active  jobs.  Consider,  for  evample,  a  two  proces¬ 
sor  system  with  job  i  requiring  repetitive  cycles  of  50  milliseconds  from  proces¬ 
sor  1,  followed  by  100  milliseconds  from  processor  2..  Under  a  light  system 
demand,  the  job  may  go  through  ten  cycles  during  a  three  second  interval, 
receiving  0.5  seconds  of  service  time  from  processor  1  and  1.0  second  of  service 
time  from  processor  2.  Under  a  heavy  demand,  within  three  seconds,  the  same 
job  could  have  made  two  cycles  only,  receiving  0.1  seconds  of  service  time  from 
processor  1  and  0.2  seconds  of  service  time  from  processor  2.  In  both  cases,  the 
ratios  of  the:  amounts  of  processing  received  are  the  same,  6n  =  1.0  and 
=  2.0,  but  the  amounts  of  processing  per  time  unit  vary  by  a  factor  of  five. 
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Because  jobs  might  he  monitored  while  being  part  of  different  service  sets, 
it  is  more  accurate  to  compare  demands  of  jobs  on  the  basis  of  the  ratios  of  the 
processing  received,  i.e.  their  intrinsic  demands,  rather  than  on  the  basis  of  the 
demand  received  under  competition.  In  preemptive  systems,  where  monitored 
information  is  used  as  input  for  the  macro-scheduler,  the  ratios  of  intrinsic 
demands  can  be  more  accurately  predicted  than  the  demands  under  competi¬ 
tion,  since  the  service  set  to  which  a.  job  is  added  is,  usually,  different  from  the 
service  set  with  which  the  job  has  been  active.  Among  the  algorithms  surveyed 
in  Section  2.3,  the  schedulers  suggested  by  Mahl,  Kameda  and  Gotlieb,  and  Badel 
and  Leroudier  require  knowledge  of  intrinsic  demand  ratios,  whereas  the  SRM 
load  adjusting  algorithms  require  knowledge  of  demands  under  competition. 


BALANC  can  be  modified  for  the  case  where  processor  balance  values  are 
lower  than  1.0.  Given  processor  balance  values  equal  to  a1l...)afc,  the  point  of 
system  balance  in  the  geometric  model  would  be  redefined  as  a*),  and  the 

l-Uj  term  in  the  selection  formula  would  be  rewritten  as  cry—  Uj.  The  next 
scheduler  is  based  on  the  redefinition  of  the  point  of  system  balance. 

RELTOT,  a  variation  of  BALAMC,  operates  on  the  assumption  that  in  order  to 
achieve  high  productivity,  processor  balance  should  be  proportional  to  the 
demand  by  the  entire  load  on  the  system,  i.e.  service  set,  phis  inactive  set.  As  a 
result,  the  point  of  system  balance  in  the  geometric  model  is  redefined.  The  jth 
coordinate  of  the  point  of  system  balance  varies  in  proportion  to  Ty,  which  is  the 
total  demand  for  the  corresponding  processor  by  all  jobs  in  the  system.  The 
coordinate  of  the  processor  which  is  the  most  in  demand  is  set  to  1.0  and  other 
coordinates  are  determined  relative  to  it.  The  selection  criterion  of  RELTOT  is 
as  follows: 
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wfaere  rj  =  (  £  ay )/‘rMaz  is  the  normalized  total  demand  per  time  unit  for 
ieJ\jS 

processor  j,  and  t^,  =  Max  £  «y- 

j€K  ie  I  [jS 

Among  the  algorithms  mentioned  in  Section  2.3,  the  one  employed  by  the 
swapping  analysis  algorithm  of  SRM  (Lynch  and  Page  1974,  and  IBM  1978)  has 
been  implemented.  Unlike  the  swapping  analysis  algorithm  of  SRM  which  recom¬ 
mends  jobs  for  addition  to  and  for  deletion  from  the  service  set,  we  consider 
additions  only.  Under  the  SRM  swapping  analysis  algorithm  all  the  candidates 
for  swapping  correspond  to  the  same  domain.  Therefore^  in  the  case  of  the  SRM 
productivity  scheduler,  the  inactive  set  includes  inactive  jobs  associated  with 
the  same  domain.  The  SRM  productivity  algorithm  has  been  expanded  to  com¬ 
pute  a  recommended  value  tor  each  processor,  such  that  the  selection  formula 
is  as  follows: 

Wax  £  M1  ~  Uif 

jcK 

This  formula  leads  to  favouring  jobs  with  relatively  high  demand  for  underutil¬ 
ized  processors. 

Ideally,  a  scheduler  makes  an  optimal  selection  for  the  service  set  while 
consuming  minimum  resources,  especially  during  invocation  time.  In  order  to 
make  a  selection  which  will  minimize  system  imbalance,  the  following  procedure 
is  necessary.  Repeat  for  each  candidate  job:  activate  the  job,  process  the  new 
service  set,  record  system  imbalance,  and  compare  the  results  for  the  job  which 
produced  the  smallest  imbalance.  The  overhead  involved  in  achieving  the 
optimal  solution  makes  this  solution  unacceptable.  In  comparison  to  the  optimal 
solution,  the  heuristics  proposed  in  this  section  incur  little  overhead. 

All  the  algorithms  described  in  this  section  require  measurements  of  pro¬ 
cessor  utilization.  Also,  common  to  the  algorithms  is  the  necessity  to  compare 
all  candidates  with  respect  to  some  property.  In  total,  |/|  comparisons  are 
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required.  The  number  of  comparisons  can  be  reduced,  in  some  cases,  by 
employing  special  data  structures  such  as  multiple  lists.  The  number  of  addi¬ 
tional  operations  required  by  each  scheduler  is  described  next.  HCLUP  and 
LCMUP  require  k  comparisons  to  find  the  least/most  utilized  processor.  BALANC 
requires  the  computation  of  -  /y)  <5XT-2  for  every  candidate  x.  This' computation 

^  jeX 

includes  k—  1  additions,  k  multiplications,  and  a  square  root  calculation.  The 
computation  can  be  carried  out  prior  to  invocation  time.  At  invocation  time, 
BALANC  requires  k  subtractions  to  calculate  1— Uj  for  each  processor.  Additional 
k  multiplications,  k  —  1  additions  and  one  division  are  required  per  candidate. 
The  requirements  of  RELTOT  are  the  same  as  the  requirements  of  BALANC,  plus 
the  computations  of  the  normalized  total  demand  per  processor.  Computing 
total  demand  per  processor  is  done  by  updating  a  size  k  array  which  holds  the 
values  of  total  demand  for  processors.  The  values  in  the  array  Eire  updated  when 
jobs  enter  or  leave  the  system.  RELTOT  also  requires  monitoring  the  amount  of 
processing  received  by  active  jobs.  SRM  requires  k  subtractions  and  k  multipli¬ 
cations  to  compute  (1-Lr;)2  for  each  processor.  Additional  k  multiplications  and 
k- 1  additions  are  required  per  candidate.  The  total  number  of  operations 
required  by  the  schedulers  at  each  invocation  time  is  summarized  in  Table  2. 


schedulers 

number  of  operations 

HCLUP  and  LCMUP 

*  +  |/i 

BALANC  and  RELTOT 

k  -t  (2k  +  1)\I\ 

SRM 

2k  +  2k\I  j 

Table  2.  Number  of  operations  at  invocation  time 

A  small  amount  of  storage  is  required  for  the  calculations.  Among  the  suggested 
algorithms,  RELTOT’ s  requirements  of  k  +  \I  j  storage  locations  are  the  highest. 
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3.2.  The  Simulated  System 

To  test  the  macro-schedulers:  with  respect  to  the  short-term  objective,  a 
stochastic,  event-driven  simulation  program  was  -written  in  PL/C.  In  the  simu¬ 
lated  system  new  arrivals  join  a  pool  of  inactive  jobs.  A  constant  number  of  jobs 
is  maintained  in  the  system.  Starting  with  a  certain  number  of  jobs  present  in 
the  inactive  set,  for  every  job  which  Leaves  an  arrival  is  generated.  To  provide 
the  schedulers  with  the  information  they  require,  the  following  procedure  was 
adopted.  Every  processor  is  associated  with  a  probability  distribution  function, 
fy  Upon  arrival,  a  job  is  assigned  a  set  of  values  Cy,  one  for  every  processor, 
sampled  from  the  functions.  Cy  is  the  amount  of  service  job  i  receives  each 
time  it  is  assigned  processor  j. 

The  total  number  of  visits  a  job  makes  to  all  processors  is  fixed  for  every 
job,  and  equals  k-MULT,  where  k  is  the  number  of  processors,  and  MULT  is  the 
expected'  number  of  visits  a  job  makes  to  each  one.  This  is  the  result  of  giving 
equal  probabilities  to  all  processors  in  a  random  selection  of  the  first  and  next 
processor  requested  by  a  job.  In  other  words,  the  first  processor  a  job  visits  is 
randomly  chosen  from  among  ail  processors.  After  a  job  completes  its  service  at 
a  processor,  the  next  processor  the  job  visits  is,  again,  randomly  chosen  from 
among  all  processors.  The  sequence  of  visits  is  repeated  k-MULT  times,  before 
the  job  is  completed.  With  this  scheme,  the  expected  total  processing  demand 
of  job  i  from  processor  j  is  MULT ■  C^. 

As  mentioned  in  Chapter  One*  storage  capacity  is  not  a  constraint,  apart 
from  limiting  the  multiprogramming  level.  In  order  to  simplify  the  system,  jobs 
cannot  be  preempted;  that  is  once  jobs  are  selected,  they  remain  in  the  service 
set  to  completion.  Since  the  multiprogramming  level  is  fixed  for  the  duration  of 
the  simulation,  the  macro-scheduler  is  invoked  to  select  a  new  job,  only  after  a 
job  has  terminated.  Once  a  job  becomes  active,  it  can  be  served  by  at  most  one 
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processor  at  a  time.  This  simplifies  the  simulation  and  appears  as  an  assump¬ 
tion  in  the  analytical  studies  of  macro-schedulers  mentioned  earlier.  Two 
micro-scheduling  disciplines  were  implemented:  FCFS  and  PS.  During  a  simula¬ 
tion  run  one  of  these  rules  is  applied  to  all  processors. 

The  FCFS  scheduling  discipline  was  simulated  in  addition  to  the  macro- 
schedulers  described  earlier.  As  FCFS  operates  independently  of  processor  util¬ 
ization  and  job  demands,  it  serves  as  a  comparison  scheme  for  the  other 
schedulers. 

The  stochastic  elements  of  the  simulation  are  based  on  a  PL/C  built-in  func¬ 
tion  called  RAND  which  is  a  generator  of  pseudo-random  numbers.  Any  time  a 
given  seed  is  used  with  RAND,  the  same  pseudo-random  number  is  generated. 
The  program  exploits  this  fact  by  carrying  out  the  simulation  for  a  specified 
length  of  time  with  one  macro-scheduler.  After  results  are  accumulated,  the 
program  goes  through  a  reinitialization  phase,  and  an  identical  sequence  of  jobs 
is  generated  for  a  different  macro-scheduler.  All  schedulers  are  examined  with 
the  same  data  during  one  run  of  the  program. 

3.3.  Simulation  Results 

In  one  sense  results  of  the  simulations  were  disappointing,  as  they  demon¬ 
strated  that  none  of  the  macro-schedulers  is  superior  to  the  others.  In  particu¬ 
lar,  BAiiANC,  contrary  to  intuition,,  did  not  prove  to  be  consistently  better  than 
the  simpler  heuristics  in  which  only  one  processor  is  considered.  However,  all 
five  balancing  schedulers  are  better  than  FGFS.  As  an  illustration,  results  of 
three  typical  simulations  are  presented  (see  Figures  2,  3,  and  4).  Parameters 
common  to  all  three  examples  are: 

*  number  of  processors  k  =  3 

•  expected  number  of  visits  a  job  makes  to  each  processor  MULT  -  10 
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«  size  of  service  set  s  =  10 

*  a  processor-sharing  micro-scheduling  discipline  for  all  processors 

The  simulations  were  carried  out  for  the  duration  of  40,000  units  of  simulated 
time  for  each  macro-scheduler.  The  average  of  processor  utilizations  since  the 
first  invocation  of  the  scheduler  was  calculated  at  regular  intervals  and  plotted 
accordingly. 

Our  purpose  in  presenting  the  following  examples  is  to  demonstrate  the 
short-term  behaviour  of  the  proposed  macro-schcdulcrs.  Some  confidence  in 
the  results  can  be  gained  by  calculating  the  number  of  job  completions,  which 
equals  the  number  of  scheduling  intervals  included  in  each  run.  In  the  first  two 
examples,  a  job  requires  on  the  average  200  time  units  from  each  processor. 
Within  40,000  time  units,  about  200  jobs  are  completed  provided  there  is  no  pro¬ 
cessor  idle  time.  Since  the  processors  are  idle  for  about  15%  of  the  time,  we 
expect  the  number  of  completions  to  be  about  15%  less  than  200,  i.e.  170.  The 
approximated  number  of  scheduling  intervals  included  in  the  third  example  is 
87.  More  confidence  in  the  results  can  be  gained  from  the  fact  that,  following  an 
initial  stage  of  instability,  variations  in  the  results  were  limited  to  about  1.0  per¬ 
centage  point.  Because  the  simulations  proved  expensive,  confidence  intervals 
were  not  computed  for  results  concerning  the  short-term  objective.  In  Section 
5.3,  confidence  intervals  are  calculated  for  the  final  results,  concerning  the 
long-term  objective. 

In  the  first  example,  the  ina.ctive  set  consists  of  50  jobs.  The  fj  functions 
from  which  the  mean  of  the  demand  for  a  processor  is  sampled  are  equal  for  all 
processors,  f  -L  =  f  z  =  / 3  =  Uniform  (0,40).  figure  2  depicts  the  results  from 
which  it  is  suggested  that  LCMUP  is  better  than  all  other  macro-schedulers, 
improving  average  utilization  over  FCFS  by  about  5  percentage  points. 
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In  the  second  example  (Figure  3),  all  parameters  remain  constant  except 
for  the  size  of  the  inactive  set  which  is  increased  to  500  jobs.  In  this  case,  the 
macro-schedulers  have  a  tenfold  larger  selection  of  candidates  to  choose  from. 
As  a  result,  the  improvement  in  average  utilization  over  FCFS  is  by  about  7 
points.  Also,  the  order  of  the  macro-schedulers  according  to  efficiency  has 
changed;  BALANC  and  RELTOT  are  best  here.  Note  that,  as  expected,  because 
the  demand  functions  for  all  processors  are  equal  and  the  number  of  jobs  in  the 
system  is  large,  the  performances  of  RELTOT  and  BALANC  are  close.  In  addition, 
typical  of  the  situation  when  the  inactive  set  is  large,  the  efficiency  of  the 
macro-schedulers  declines  with  time.  The  reason  is  that  the  variety  of  candi¬ 
dates  decreases  when  thev  are  cont.inuousiv  subjected  to  one  tvoe  of  selection. 

J  a*  M  w  a 

For  instance,  when  HCLUP  is  continuously  active,  the  highest  consuming  jobs  are 
removed  from  the  inactive  set,  decreasing  the  variance  among  the  remaining 
jobs.  With  an  initial  increased  size  of  the  inactive  setT  the  change  of  the  variance 
within  the  set  is  expected  to  become  more  pronounced.  The  decline  in  the 
efficiency  of  the  schedulers  suggests  another  reason,  in  addition  to  those  given 
in  Chapter  One,  for  using  the  same  macro-scheduler  for  a  short  time  only. 

In  the  last  example  (Figure  4),  total  demands  for  processors  are  not  equal, 
i.e.  a  bottleneck  situation  occurs.  The  size  of  the  inactive  pool  is  set  at  250  jobs, 
and  the  values  of  the  fj  functions  are  changed  to;  f\  =  Uniform  (0,10),  /2  =  Uni¬ 
form  (0,40),  and  /a  =  Uniform  (0,60).  Here,  BALANC  proves  superior.  It 
improves  average  utilizations  when  compared  with  FCFS  by  about  15  percentage 
points.  Note  that  since  total  demand  for  processors  is  unequal,  the  efficiency  of 
RELTOT  is  quite  different  from  the  efficiency  of  BALANC. 

Many  simulation  runs  were  made  in  which  the  influence  of  different  system 
parameters  on  the  performance  of  the  macro-schedulers  was  investigated.  The 
outcome  revealed  no  consistent  superiority  of  any  one  macro-scheduler,  even 
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under  fixed  conditions,  i.e.  when  only  stochastic  variations  were  allowed.  How¬ 
ever,  a  common  feature  of  the  simulations  seems  to  be  that  HCLUP,  LCMUP, 
BALANC,  RELTOT,  and  SRM  were  more  efficient  than  FCFS,  in  almost  every  case. 

The  conclusion  to  this  point  is  that  even  a  simple  heuristic  for  balancing  the 
system  increases  the  average  of  processor  utilizations  during  scheduling  inter¬ 
vals.  However,  none  of  the  strategies  examined  maximizes  short-term  average 
of  processor  demands,  i.e.  none  of  the  algorithms  minimizes  system  imbalance. 
To  find  an  optimal  scheduler  some  additional  insight  is  needed.  This  comes  from 
the  realization  that  in  the  proposed  macro-schedulers  there  is  no  adequate 
account  of  the  effect  of  interactions  among  active  jobs  on  processor  utilizations. 
The  fact  that  jobs  apportion  or  line  up  for  the  use  of  processors  contributes  to 
system  imbalance.  This  fact  should  influence  the  decisions  of  balancing 


schedulers. 
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CHAPTER  FOUR 


An  Optimal  Balancing 
Macro-Scheduler  for  Separable  Networks 

While  examining  the  results  of  the  preliminary  approach,  it  became 
apparent  that  existing  macro-schedulers  share  a  common  deficiency;  there  is 
inadequate  account  of  the  effect  of  interactions  among  active  jobs  on  system 
performance.  Mahl  (1970)  cannot  precisely  determine  future  progress  rates  of 
jobs  given  the  demands  of  the  service  set.  Kameda  and  Gctlieb  (1977)  use  func¬ 
tions  corresponding  to  processors;  which  map  the  priority  of  a  job  requiring  a 
unit  of  processing  to  the  amount  of  time  the  job  will  have  to  wait  for  service;  but 
when  it  comes  to  implementation,  only  a  rough  approximation  of  these  functions 
is  suggested.  Badel  and  Leroudier  (1973)  limit  the  scope  of  the  problem  by 
assuming  that  mean  processor  service  times  are  known.  In  the  system 
resources  manager  (SRM)  of  IBM  055/VS 2  MVS  operating  system  (Lynch  and  Page 
1974,  IBM  1973)  swapping  decisions  are  made  according  to  recommended  values 
which  are  computed  for  each  pair  of  processor  and  candidate.  A  recommended 
value  is  proportional  to  the  demand  of  the  job  and  to  the  square  of  the  distance 
between  current  processor  utilization  and  processor  balance.  The  job  with  the 
highest  sum  of  recommended  values  for  all  processors  is  recommended  for  addi¬ 
tion  to  the  service  set.  Still,  there  is  no  guarantee  that  such  a  scheme  results  in 
the  best  selection  of  jobs  for  the  service  set. 

For  similar  reasons,  the  algorithms  which  are  proposed  and  simulated  in 
Chaplex-  Three  ax  e  inadequate.  HCLUP  and  LCMUP  base  their  selection  of  a  job 
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for  the  service  set  on  the  utilization  of  only  one  processor.  As  a  result,  the 
operation  of  these  algorithms  might  shift  the  imbalance  instead  of  improving  it. 
increasing  total  imbalance  in  the  system.  Also  in  the  model  which  underlies  the 
BALANC  and  RELTOT  schedulers,  interactions  among  active  jobs  are  not  fully 
considered.  The  direction  in  which  the  system  operating  point  will  move  cannot, 
in  general,  be  calculated  simply  by  adding  the  intrinsic  demand  ratios  of  a  new 
job  to  the  current  system  operating  point.  Such  a  calculation  could  produce 
accurate  results  if  there  were  no  queueing  or  sharing  of  processors.  In  reality, 
however,  introducing  a  new  job  into  the  service  set  will  not  only  affect  the  con¬ 
sumption  of  this  job  but,  due  to  competition  for  the  use  of  processors,  will 
influence  the  consumption  of  other  active  jobs,  moving  the  system  operating 
point  in  an  unknown  direction. 

A  superior  balancing  scheduler  should  be  able  to  evaluate  the  effect  of  load¬ 
ing  a  new  job  into  the  service  set,  To  tackle  this  problem,  the  following  pro¬ 
cedure  is  suggested: 

1.  assume  certain  properties  of  the  system;  use  a  model  to  represent  aspects 
relevant  to  macro-scheduling, 

2.  validate  the  model,  and 

3.  analyze  it  by  solving  for  processor  utilization  given  the  characteristics  of  the 
service  set  and  the  demands  of  a  new  job. 

Provided  computations  are  fast,  the  macro-scheduler  can  check  the  effect 
on  system  balance  of  loading  each  candidate  and  choose  the  job  which  brings 
the  system  the  closest  to  a  balanced  state.  The  task  of  designing  a  fast  solution 
model  based  on  reasonable  assumptions  and  validating  it  seems  formidable. 
Fortunately,  queueing  network  models  are  suitable  for  this  purpose.  They  can 
represent  relevant  aspects  of  the  system  and,  given  the  .load  on  the  system  and 
conditioned  on  necessary  assumptions,  they  can  be  analyzed  quickly.  Further- 
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more,  queueing  network  models  have  been  successfully1  employed  as  relatively 
fast  and  accurate  tools  for  prediction  and  evaluation  of  performance.  The  first 
successful  application  of  queueing  networks  for  the  analysis  of  computer  system 
performance  has  been  reported  by  Scherr  (1967).  For  recent  applications,  see 
papers  included  in  the  proceedings  of  Performance  30  (1930). 

In  this  chapter,  analytic  queueing  network  modelling  is  applied  to  the  prob¬ 
lem  of  macro-scheduling  for  system  balance.  First,  a  concise  survey  of  queueing 
network  modelling  will  be  presented  followed  by  the  solution  method  of  mean 
value  analysis.  Next,  the  problem  of  macro-scheduling  for  system  balance  will 
be  modelled  with  queueing  networks.  Applying  mean  value  analysis  to  the 
model,  system  demand  is  expressed  as  a  recursive  function  of  the  demands  of 
one  job,  while  the  demands  of  other  active  jobs  are  the  parameters.  One  conse¬ 
quence  of  this  functional  dependence  is  a  theorem  stating  that  system  demand 
depends  on  the  ratios  of  the  intrinsic  demands  of  active  jobs.  A  second  result  is 
the  design  of  OPT,  a  macro-scheduler  which  is  based  on  the  formula  for  max¬ 
imum  system  demand  and  is  optimal  under  the  assumptions  of  the  model.  Pos¬ 
sible  generalizations  of  OPT  are  examined  followed  by  a  description  of  the  opera¬ 
tion  of  OPT  in  some  simple  cases.  A  closed-form  expression  for  the  selection  for¬ 
mula  is  introduced.  Finally,  the  computational  complexity  of  the  selection  for¬ 
mula  and  an  approximation  for  the  case  of  a  large  service  set  are  discussed. 

4.1.  An  Overview  of  Mean  Value  Analysis  of  Closed  Queueing  Network  Models 

Closed  queueing  networks  have  been  extensively  employed  for  analytic 
modelling  of  the  service  set  in  multiprogrammed  computer  systems.  A  closed 
queueing  network  is  one  which  contains  a  fixed  number  of  jobs.  Surveys  of  ana¬ 
lytic  queueing  network  modelling  including  references  are  given  in  Graham 
(1978).  and  in  Kienzle  and  Sevcik  (1979).  In  these  models,  the  system  is  viewed 
as  a  Markovian  queueing  network,  which  implies  the  following  assumptions: 
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1.  The  system  is  modelled  by  a  stationary  stochastic  process. 

2.  Jobs  are  stochastically  independent. 

3.  Job  steps  from  processor  to  processor  follow  a  Markov  chain. 

4.  The  system  is  in  stochastic  equilibrium. 

5.  Service  time  requirements  at  each  processor  are  exponentially  distributed. 

6.  The  system  is  ergodic  -  i.e.f  long-term  time  averages  converge  to  the  values 

computed  for  stochastic  equilibrium. 

A  Markovian  network  was  traditionally  solved  by  a  system  of  global  balance 
equations  for  the  equilibrium  state  probabilities.  Jackson  (1963),  Gordon  and 
Newell  (1987),  and  Baskett,  Chandy,  Muntz  and  Palacios  (1975)  discovered  that 
for  certain  classes  of  networks  called  separable,  the  solution  of  the  balance 
equations  has  a  product-form,  in  which  each  term  corresponds  to  a  processor. 
Processors  in  separable  networks  have  one  of  the  following  micro-scheduling  dis¬ 
ciplines  and  service  time  requirements: 

1.  FCFS  with  service  time  requirements  which  are  exponentially  distributed  and 
class*  independent. 

2.  Processor-sharing  with  general,  class  dependent  service  time  requirements. 

3.  Last-come-first-served-preemptive-resume  with  general,  class  dependent  ser¬ 
vice  time  requirements. 

4.  No  queueing  (NQ)  with  general,  class  dependent  service  time  requirements. 
(In  the  NQ  discipline,  all  jobs  which  require  service  from  a  processor  are  served 
in  parallel.) 

Product  terms  in  the  solution  of  separable  networks  have  to  be  normalized 
to  form  a  proper  probability  distribution.  In  the  case  of  closed  networks  with 
multiple  classes,  computing  the  normalizing  factor  requires  special  convolution 
techniques.  The  first  convolution  algorithm  was  developed  by  Buzen  (1973)  for  a 


*  A  class  is  defined  as  a  group  of  statistically  identical  jobs 
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single  class,  and  later  generalized  by  Chandy,  Herzog  and  Woo  (1975),  by  Reiser 
and  Kobayashi  (1975),  and  by  Balbo,  Bruell  and  Scnwetman  (1977)  for  multiple 
classes. 

A  second  method  for  solving  closed  separable  networks  is  mean  value 
analysis  (Reiser  and  Lavenberg  1980).  In  mean  value  analysis,  unlike  the  convo¬ 
lution  techniques,  it  is  not  necessary  to  compute  a  normalizing  factor  in  order 
to  express  relations  among  the  means  of  performance  measures.  For  a  multiple 
class  closed  queueing  network  which  has  a  product-form  solution,  Reiser  and 
Lavenberg  developed  a  recursive  analysis  relating  mean  waiting  time  to  mean 
queue  size.  Their  result  concerning  load  independent  processors  is  expressed 
by  the  formula: 

7’w{S)  =  [l+JVy(S-i)]*v  (1) 

where: 

Tij(S)  =  equilibrium  mean  total  waiting  time,  including  service  time,  of  job 

i  at  processor  j  with  service  set  S' 

tij  =  total  service  time  (intrinsic  demand)  required  by  job  i  from  processor 

j 

Nj(S )  =  equilibrium  mean  queue  size  at  processor  j  with  service  set  S 

(S  —i)  refers  to  set  subtraction 

An  explanation  of  the  above  formula  is  provided  by  the  following  theorem:  a 
class  i  job  arriving  at  processor  j  in  the  system  Q(S)  "sees"  the  system  Q(S-i) 
in  equilibrium  (Reiser  and  Lavenberg  I9B0,  see  also  Sevcik  and  Mitrani  1979). 
Equation  1  states  that  the  mean  time  a  job  will  wait  at  a  processor  equals  its  own 
mean  service  demand,  plus  the  mean  backlog  upon  arrival. 

Approximate  solutions  for  nonseparable  queueing  networks  which  involve 
mean  value  analysis  were  suggested  by  Bard  (1979)  to  handle  FCFS  processors 
with  class  dependent  service  distributions,  priorities,  overlap  among  processors, 
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blocking  due  to  unavailability  of  processors  held  by  other  jobs,  and  decompos¬ 
able  networks. 

4.2.  Modelling  and  Analysis 

The  assumption  of  stochastic  equilibrium  during  a  scheduling  interval  is  not 
severe,  since  micro-level  activities  are  many  and  fast  in  comparison  t  >  macro¬ 
level  ones.  See  Courtois  (197?)  for  an  analysis  of  decomposability  in  computer 
systems.  A  suggested  model  for  the  behaviour  of  a  muitiprogrammed  computer 
system  during  a  scheduling  interval  is  a  closed,  separable,  multiple  class  queue¬ 
ing  network.  As  jobs  in  the  service  set  may  differ  in  their  demands,  each  job  in 
the  model  is  considered  to  be  in  a  class  of  its  own.  Consequently,  the  number  of 
classes,  s,  equals  the  number  of  active  jobs,  i.e.  the  multiprogramming  level  is  s. 
In  this  scheme,  the  problem  of  macro-scheduling  for  system  balance  can  be 
stated  as  follows:  given  service  set  S',  an  inactive  job  x  has  to  be  chosen  such 
that  the  combined  new  set,  S  '^jx,  will  maximize  system  demand  P(S  \jx)  for  the 
next  scheduling  interval. 

First,  an  expression  for  P(S\Jx)  is  derived.  System  demand  was  defined 
as: 

P(S)  =  4  £  Uj(s) 

K  jeK 

where: 

K  =  the  set  of  processors,  k  -  \K\ 

Uj{S )  =  equilibrium  mean  utilization  of  processor  j  with  service  set  51 
Uj(S)  can  be  expressed  as  a  sum  of  throughputs: 

Uj(S)  =  £  KiiS'jty 

ieS 

where  \{S )  is  the  equilibrium  mean  throughput  of  class  (job)  i  with  service  set 
5\  Little’s  formula  (1961)  with  one  job  per  class  says: 

1 

Ri(s) 


A dS)  = 


(2) 
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where: 

R^S  )  =  equilibrium  mean  response  time:  of  class  i  with  service  set  S 
From  the  above  equations  it  follows,  that: 


P(S)  =  ±  2  E 


'V 


=  T  £ 


s  % 

3<eK 


k  jzK  tzS  ^  £c5  Ri(S) 

Denoting  total  demand  of  job  i  for  all  processors  by  Dif  i.e.  D * 


-  S  tiS: 

jeK 


P(S)  = 


JL  y 

k  fyg  RdS) 


The  above  equation  states  that  system  demand  is  proportional  to  the  sum 
over  all  active  jobs  of  the  ratios  of  a  job's  total  processing  demand  to  its 
response  time.  Therefore,  adding  an  extra  job,  x,  to  the  service  set  results  in 
the  following  demand: 


P(SU*) 


D, 


+  £ 


D. 


Rz(S{Jx)  £s  Ri(S{jx) 


-l 


(3) 


Next,  mean  value  analysis  is  applied  to  express  system  demand  as  a  func¬ 
tion  of  the  service  requirements  of  job  x,  while:  the  service  requirements  of  other 
active  jobs  are  considered  as  parameters.  Response  time  is  composed  of  waiting 
times  plus  processing  times  at  all  processors,  i.e.: 


XdSijx)  =  £  Tv(S\jx) 

jc-K 

From  mean  value  analysis  (see  Equation  l): 


TqiS)  =  [1  +  Nf(S -i)]  tq 

Letting  N^(S)  denote  the  equilibrium  mean  queue  size  of  job  i  at  processor  j 
with  service  set  S,  equilibrium  mean  queue  size  at  a  processor  with  respect  to 
one  class  can  be  computed  according  to  Little’s  Law: 


Ni;(S)  =  \(S) 


T 

M  ij  J 


Applying  Equation  2  and  summing  over  ail  jobs: 


Nj(S) 


y  7V,<S) 

h  RdS) 


(4) 
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Equations  1  and  4  establish  a  recursive  solution  to  the  response  time  of  job  i: 

Ri(S  \jx)  =  X!  Ti}(S  \jx)  =  X)  [1  +  Nj(S  kj  ~  A  +  E  U*”^) 

jcK  jeK  jzK 

The  above  equation  means  that  the  expected  response  time  of  job  i  equals  its 

total  processing  demand  plus  a  weighted  sum  of  the  equilibrium  mean  number 

of  jobs  at  each  processor.  Equilibrium  mean  number  of  jobs  at  a  processor  is 

calculated  for  a  system  which  differs  from;  the  original  one  by  having  job  i 

removed.  Two  instances  are  of  interest: 

1.  i  =  x 


2.  i  *  x 


Rx(S\Jx)=Bx+  2  t^Ns(S) 


(5) 


„  „  Tci(S{jx-i) 

Ri(S  \jx)  =  Di  +  XI  kg  '  S  ze  (si  j x-i) 

Splitting  the  sum  over  classes  results  in  the  following  equation: 


Ri{S  iji)  =  Di  + 


\*z 


E  kf  Tzj(s  U x “i )  E  ki  T v (s  Uz-i) 

y\  jsK _ 

Rx(Sijx-i)  oclLf  Re(S\jx-i) 


Applying  Equations  1  and  5  to  the  second  term: 

2  hj  hi  [1  +  AT,(S-i)] 
Rj(S  <jx  )  =  Dt  +  tz.  Nj(S-i) 

j*K 


E  kj  Tcj (S  {Jx  i) 

+  y  - (6) 

c&-*  Rc{S''jx-i)  {  } 


Stopping  criteria  for  the  recursion  are  reached  when  only  one  job,  b,  remains  in 
the  service  set.  The  following  equations  hold  for  the  last  job: 

TbJ(b)  =  tb3  and  Rb(b)  =  X  )  =  E  hi  =  A 

je.K  j*K 

System  demand  can  now  be  expressed  in  terms  of  job  demands  by  substi¬ 
tuting  Equation  5  in  Equation  3: 


P(Syx) 


A 


A  +  E  tzj 


+  E 

ieS 


Di 


Rt{S\jx) 


i--i 


where  Ri(S  \ jx)  is  defined  in  Equation  8. 


(7) 
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Equation  7  with  the  recursive  formulae  1  and  4  can  be  employed  by  a 
macro-scheduler  to  maximize  system  demand  during  the  next  scheduling  inter¬ 
val.  The  following  scheme  is  suggested:  for  a  given  number  of  processors,  a 
group  of  functions  P(S  ijx)  can  be  defined  for  a  range  of  multiprogramming  lev¬ 
els.  The  macro-scheduler  will  choose  a  function  according  to  current  multipro¬ 
gramming  level,  assign  the  demands  of  active  jobs  to  the  parameters,  scan  all 
candidates,  and  select  the  one  whose  service  demands  maximize  P(S\jx). 
Before  considering  the  operation  of  such  a  macro-scheduler,  a  theorem  con¬ 
cerning  the  type  of  information  it  requires  is  developed. 

4.3.  Input  for  the  Macro-Scheduler 

In  order  to  balance  the  system,  information  about  processing  demands  of 
the  candidates  for  addition  to  the  service  set  is  required.  As  mentioned  in  Sec¬ 
tion  3.1,  it  is  simpler  to  compare  jobs'  processing  demands  on  the  basis  of  their 
intrinsic  demands,  than  on  the  basis  of  their  demands  under  competition.  Also, 
predictions  of  ratios  of  intrinsic  demands  are  easier  than  predictions  of 
demands  under  competition.  This:  is  the  significance  of  the  following  theorem 
which  proves  that  in  order  to  balance  the  model  of  the  system  the  only  required 
information  is  that  of  the  ratios  of  a  job’s  intrinsic  demands.  First,  a  lemma  is 
proved. 

Lemma.  Given  the  service  set  5  ijx,  where  5  is  a  parameter  and  x  is  a 
designated  job,  P(S  IJx)  is  determined  by  the  ratios  of  the  intrinsic  demands  of 
job  x. 

Proof.  Let  x  be  associated  with  a  vector  of  service  demands  L.  =  tTk  ).  The 

lemma  says  that  P^S'^Jx)  is  determined  by  a  vector  ~&x  -  5^),  where 

=  tXj/ixq  for  all  je  K  and  some  prespecified  processor  q,  q  e  K. 

Dividing  both  the  numerator  and  the  denominator  in  the  first  term  of  Eaua- 
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tion  7  by  tzq  results  in: 


P{S^x)  = 


£  <5xy 

_ &L _ +  y  _ _ 

£  ^  +  £  <5*;  N,IS)  ffs  Pi (S  U*) 


.-i 


As  the  first  term  above  is  determined  by  t$T,  it  remains  to  show  that  so  is 

Pi  (5'jr).  From  Equation  8.  after  dividing  the  numerator  and  the  denominator 
irz 


of  the  second  term  by  Ltq,  it  follows  that: 

E  %  <Sx>  [1  +  Ns(S-i) ] 
Ri(S^x)  =  Dt+  l- 


£  Uj  Tcj(S  l ,'z-i) 


trz 


jetf  ;eif 

Applying  Equations  1  and  4: 


- +  y  UK - 

£  Xj(s-i)  e£-i  /2c(Su*-0 


T’cyC-S  Ux— i)  =  tcj 


Taj(S'jz-i-c) 
_Cl,  .  /?a(^  ljx-i-c) 

a£5'Ji-«-C  a\  w  ' 


Keeping  the  recursion  on  Taj(S  '\Jx  —  i -c )  until  all  classes  in  S’  but  one,  6.  are 
exhausted  results  in: 


1  bj(P  \J ~~  L bj 


1  + 


Tyjx) 

Rr(x) 


=  h,  (1+  If-)  =  tbj  (1  + 


D 


E^ 

;eJf 


) 


Hence,  the  recursive  formula  for  Ri(S  \Jx)  is  determined  by  <$x.  The  recur- 


l*X 


sion  stops  when  only  two  classes,  b  and  x,  remain  in  the  service  set,  for  which 
Equation  6  is  reduced  to: 


£  tbjtxj 


Pb(b  \JX  )  -  &h  + 


5*K 


Dz 


=  £*  + 


£  hj^zj 

Y  6~<  ' 

t—t 

izK 


QED 


Theorem.  For  a  given  service  set  S',  P(S')  is  solely  determined  by  the  ratios  of 
the  intrinsic  demands  of  each  job  in  5". 

Proof.  Appl}'  the  above  lemma,  in  turn,  for  each  job  linS',  such  that  5"  =  51  ‘\jx. 


QED 
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4.4.  The  Operation  of  OPT 

The  macro-scheduler  described  in  Section  4.2  will  be  referred  to  as  OPT.  It 
should  be  stressed  again  that  OPT’s  optimality  is  conditioned  on  the  assump¬ 
tions  of  separable  queueing  networks  (see  Section  4.1)  and  is  limited  to  one 
scheduling  interval  only. 

OPT  can  be  adapted  for  general  processor  balance  points.  Given  ak  as 

the  values  of  processor  balance,  the  short-term  objective  of  the  algorithm  would 
be  to  minimize  system  imbalance: 


7  £  ! ai  ~  Uj(s \jx)\  =|S!«j-  E  rWT^) 

k  jzK  k  jeFC  teSU*  ^ * 

where  Ri(S  yx)  is  defined  in  Equations  5  and  6. 


If  weighting  of  processors  is  required,  assume  Wj  is  the  weighting  of  proces¬ 
sor  j,  then: 


WjUjiSlJx)  =  Wj  J] 


ks 


E  'wjkj 


nfn,.  r  and  E  /_  v 


The  only  change  required  in  the  selection  formula  of  the  scheduler  is  making  the 

numerator  terms  of  Equation  7  instead  of  Dit  for  all  51  yx. 

jzK 

Balancing  is  sometimes  necessary  for  just  a  subset  of  all  the  processors  in 

the  system.  Let  K  denote  the  set  of  all  processors  which  affect  jobs’  response 

time,  and  let  K'  denote  the  set  of  processorsrfor  which  balancing  is  desired.  By 

changing  the  numerator  terms  in  Equation  7  from  Di  to  VJ  for  all  i€  S  yx, 

PK’ 


the  scheduler,  would  be  balancing  the  processors  of  set  K'  only. 


In  the  proof  of  the  theorem  in  Section  4.3,  it  has  been  shown  that 


D< 


Ri( S ) 


for 


any  job  i  in  <5*  is  determined  by  the  ratios  of  the  intrinsic  demands  of  each  job  in 
S.  This  is  true  if  the  Dt  in  the  numerator  is  replaced  by  any  linear  combination 
of  tij's.  Consequently,  the  theorem  of  Section  4.3  holds  for  the  case  of  different 
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values  of  processor  balance,  for  the  case  of  weighted  processors,  and  for  the 
case  of  balancing  a  subset  of  processors. 

Four  simple  examples  of  formula  7,  indicating  the  operation  of  OPT,  appear 
in  Appendix  A.  From  these  examples  we  conclude  that  balancing  in  a  two  pro¬ 
cessor-  system  can  be  dune  by  checking  the  utilizations  of  the  two  processors.  If 
processor  1  is  more  utilized  than  3,  then  a  candidate  x  with  the  highest  ratio  of 
txz/tx\  be  recommended  for  loading.  If  the  reverse  is  true,  then  a  candidate 
is  chosen  such  that  tx z/tx\  is  minimum.  If  both  processors  are  equally  utilized, 
any  job  can  be  loaded.  In  the  case  of  three  processors,  the  objective  function 
may  fluctuate  with  an  increase  in  tT,z/tT\  or  with  an  increase  in  tx^/tx^.  The 
overall  approach  for  maximizing  system  demand  is  to  substitute  the  intrinsic 
demand  ratios  of  each  candidate  in  the  formula  for  system  demand,  and  to 
select  the  candidate  which  yieids  the  highest  value. 

If  the  recursive  formula  of  OPT  could  be  inverted  to  allow  calculation  of  the 
"perfect"  balancing  job,  a  search  for  the  closest-to-perfect  job  could  be  carried 
out  using  a  k-d  tree  data  structure  to  decrease  the  number  of  operations.  How¬ 
ever,  the  recursion  cannot  be  inverted  since  system  demand  depends  on 
Ri(S  {jx),  which  is  recursively  computed  according  to  Equation  8. 

\¥-Z 

4.5.  A  Closed-Form  Selection  Formula 

Given  a  certain  multiprogramming  level,  s,  and  a  specific  number  of  proces¬ 
sors,  k,  the  set  of  recursive  formulae  1  and  4-7  can  be  rearranged  by  algebraic 
manipulations  as  a  closed-form  expression  for  system  demand.  Examples  of 
closed-form  expressions  for  multiprogramming  levels  between  two  and  five 
appear  in  Appendix  B.  General  characteristics  of  the  closed-form  solution  are 
described  hereafter.  System  demand  is  a  ratio  of  two  product-form  expressions 
which  are  symmetrical  with  respect  to  all  active  jobs.  A  product  term  is  made 
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up  of  s  values,  one  for  each  active  job.  A  value  can  either  be  the  total  demand  of 
a  job,  Da,  or  the  job’s  demand  for  a  processor,  taj.  If  taj  appears  in  a  term,  then 
it  is  part  of  the  multiplied  demands  of  at  least  two  jobs,  summed  over  all  proces¬ 
sors,  Le.  E  All  terms  in  the  numerator  include  one  Da  or  more,  whereas 

j  cK 

the  denominator  contains  some  terms  with  no  Da' s.  The  number  of  terms  in  the 
expression  for  system  demand  grows  factorially  with  the  multiprogramming 
level.  This  is  due  to  the  fact  that  in  order  to  compute  the  response  time  of  some 
job  i,  the  response  times  of  ail  other  jobs  in  the  service  set  have  to  be  computed 
when  job  i  is  removed  (see  Equation  6). 

Product  terms  which  appear  in  the  numerator  also  appear  in  the  denomina¬ 
tor,  possibly  with  different  coefficients.  The  coefficients  are  governed  by  the  fol¬ 
lowing  rules: 

1.  A  coefficient  is  equal  to  one. 

2.  For  every  summation  (E)  which  appears  in  the  term,  its  coefficient  is  mul¬ 
tiplied  by  the  factorial  of  the  number  of  elements  associated  with  the  sum¬ 
mation,  minus  one. 

3.  For  numerator  terms  only,  the  coefficient  is  also  multiplied  by  the  number 
of  Z?a’s  in  the  term 

In  order  to  minimize  the  amount  of  computation  performed  during  invoca¬ 
tion  time,  the  formula  for  system  demand  can  be  written  as: 

a ’At  +  E  bj'txj 
_ _ 

c-Dx  +  J]  di'ixj 

fear 

where  the  coefficients  a,  bj,  c,  and.  dj  are  independent  of  x  and  computed  prior 
to  invocation  lime,  if  possible.  The  number  of  operations  needed  to  calculate 
the  coefficients  for  a  multiprogramming  level  of  three  is:  fc(fc-l)+3  for  a,  3  for 
&y,  k(k  — 1)+2  for  c,  and  6  for  dj.  Additional  2{k  —  i)  operations  are  required  to 


P(S  u*>  = 
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calculate  Da  and  Db.  The  growth  of  the  number  of  operations  is  factorial  in  s, 
but  some  repeated  calculations  can  be  avoided  by  storing  intermediate  results. 
Once  a,  bj,  c,  and  dj  are  computed,  the  number  of  operations  required  to  evalu¬ 
ate  a  candidate  is  2k +2  multiplications,  2 k  additions,  and  one  division.  If  the 
multiprogramming  level  is  such  that  the  amount  of  computation  for  determining 
the  coefficients  is  limiting,  approximations  should  be  investigated.  For  instance, 
an  approximation  has  been  developed  based  on  the  following  observations 
regarding  the  optimal  solution: 

1.  The  dominant  term  in  the  numerator  is  sDxY\P%-  Other  terms  can  either  be 

i£S 

combined  to  the  form  DXY\P\*  or  else  they  are  discarded. 

ieS 

2.  Two  distinct  terms  in  the  denominator  are  Aeli^t  and  (s  — 1)!  X!  Any 

icS  jeff  ieS" 

other  term  is  a  combination  of  D^'s  and  ty's,  therefore  it  is  discarded. 

The  result  is  the  following  selection  criterion: 


jeX  ieS 

APP  is  a  scheduler  based  on  the  above  approximation  formula.  The  approxima¬ 
tion  is  identical  to  the  optimal  formula  in  the  simple  cases  discussed  in  Appen¬ 
dix  A  (to  be  analyzed  in  Section  5.2). 


schedulers 

number  of  operations 

OPT 

(4fc  +  3)|/ I 

APP 

2k\I\ 

Table  3.  Number  of  operations  at  invocation  time 


APP  requires  .only  sfc+s+fc- 1  arithmetic  operations  to  produce  a  size  k  vector 

of  These  operations  can  be  performed  prior  to  invocation 

icS  icS 
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time,  such  that  when  the  macro-scheduler  is  invoked,  2 k  arithmetic  operations 
are  required  to  evaluate  a  candidate.  The  number  of  operations  required  by  OPT 
and  by  APP  at  each  invocation  time  is  summarized  in  Table  3.  The  performances 
of  APP  and  other  macro-schedulers  will  be  studied  in  the  next  chapter. 
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CHAPTER  FIVE 


Comparative  Studies  of  Macro-Schedulers 

In  Chapter  Three,  simulation  studies  of  five  macro-schedulers  for  system 
balance  were  examined.  Results  indicated  that  in  the  majority  of  cases  they 
performed  better  than  FCFS,  a  nonbalaxicing-  scheduler,  improving  the  average 
of  processor  utilizations  by  as  much  as  15  precentage  points.  Another  outcome 
was  that  none  of  the  schedulers  proved  to  be  consistently  superior  to  the  others. 
Different  simulation  runs  in  which  the  distribution  of  job  demands  remained 
fixed,  allowing  only  statistical  variations,  resulted  in  a  changed  order  of 
efficiency  among  the  schedulers.  Conclusions  were  that  the  tested  algorithms 
improve  the  average  of  processor  utilizations r  but  the  problem  of  finding  a  supe¬ 
rior  balancing  scheduler  remained  unsolved.  In  order  to  choose  an  appropriate 
scheduler,  the  influence  of  system  workload  on  the  performance  of  the 
schedulers  has  to  be  understood.  In  this  chapter,  an  attempt  is  made  to  explain 
the  performance  of  the  schedulers  in  the  framework  of  the  model  developed  in 
Chapter  Four.  First,  the  selection  criteria  of  the  macro-scheduling  algorithms 
are  compared  with  the  formula  used  by  OPT.  Next,  the  performance  analyses  of 
the  macro-schedulers  are  generalized  and  their  efficiency  studied  with  simula¬ 
tion.  Last,  the  long-term  effect  of  the  macro-schedulers  on  productivity  is 


examined. 
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5.1.  An  Analytic  Comparison 

An  insight  into  the  performance  of  HCLUP,  LCMUP,  BALANC,  RELTOT,  SRM, 
and  APP  is  gained  by  applying  them  to  the  examples  of  Appendix  A.  Ail  solutions 
to  Example  1  are  optimal,  so  that  Example  2  is  considered  first.  In  Example  2 
there  arc  two  processors,  K  -  \  1,  2\,  and  one  active  job,  a,  such  that  processor 
utilizations  are  U i{S )  -  ta  \/Da  and  Uq(S  )  -  taz/Da.  Table  4  summarizes  the 
criteria  for  selecting  a  second  active  job.  The  efficiency  of-  the  macro¬ 
schedulers  can  be  compared  with  the  optimal  solution,  OPT.  HCLUP  and  LCMUP 
will  not  produce  optimal  results,  because  their  selection  is  based  on  demand  for 
one  processor  only.  Nevertheless,  in  most  cases  they  will  perform  better  than  a 
nonbalancing  algorithm  because,  like  OPT,  their  selection  is  adaptive  to  changes 
in  congestion,  and  for  a  single  processor  their  criteria  are  best. 


case  1 

case  2 

case  3 

schedulers 

ta  1  >  ta2 

ta  l  <  ta 2 

ta  1  =  ta2 

OPT  and  APP 

tx  2 

Max 

*xl 

tx  i 

Max 

tzZ 

any  job  x 

HCLUP 

Max  tx  2 

Max  tx  i 

any  job  x 

LCMUP 

Min  txl 

Min  tx2 

any  job  x 

for  all  cases 

BALANC 

txZ  i 

ta.2 

RELTOT 

txZ  ~ 

tx\  T\^a  tai 

SRM 

Max  (tx  1^0,2  ) 

Table  4.  Selection  criteria  for  two  processors  and  one  active  job* 


The  operation  of  BALANC  is  also  adaptive.  The  ideal  candidate  x  chosen  by 


•  The  notation  X  -»  (X  means  the  value  of  X  which  is  the  closest  to  (X  among  all  possible 
values  of  X. 
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BALANC  would  have  a  ratio  £*2/^1  which  is  greater  than  one  if  £al  >  taZ.  If 
ta  1  <  fa8,  then  the  ratio  tx2/tx\  of  the  ideal  candidate  would  be  less  than  one. 
This  criterion  produces  better  results  than  a  nonbalancing  scheduler,  and 
improves  with  greater  inequality  in  processor  utilizations,  i.e.  with  larger  values 
of  tai/taZ.  When  the  processors  are  utilized  to  about  the  same  level,  the  cri¬ 
terion  of  BALANC  is  far  from  the  optimum,  but  in  this  case,  the  effect  which  a 
new  job  can  have  on  system  demand  decreases  (see  Table  4  case  3). 

The  performance  of  RELTOT  depends  on  total  demand  in  the  system.  If  the 
demand  for  the  two  processors  is  about  the  same,  then  the  performance  of  REL¬ 
TOT  will  be  close  to  the  performance  of  BALANC.  Otherwise,  the  efficiency  of 
RELTOT  depends  on  the  relation  between  the  difference  in  utilizations  and  the 
difference  in  the  demand  for  processors.  With  significant  differences  in  the 
demand  for  processors,  the  efficiency  of  RELTOT  will  not  necessarily  be  better 
than  that  of  a.nonbalanclng  scheduler. 


*x2  tx1~tx2 


Selections  of  macro-schedulers  for  the  case  of  an  equal 
total  demand  K  =  \  1,2],  S'  =  [a],  ta  1  <  ta 2 


Fig.  5. 
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SRM  will  be  more  efficient  than  a  nonbaiancing  scheduler,  because  jobs  with 
high  demand:  for  the  less  utilized  processor  me  favoured.  Since  ratios  of 
demands  are  not  considered  by  SRM,  its  performance  is  not  expected  to  be 
optimal. 

Figure  5  illustrates  the  candidates  selected  by  the  various  macro¬ 
schedulers  when  total  demand  for  processors  is  equal,  tj  =  Tz.  In  this  figure,  a 
candidate  is  represented  as  a  point  according  to  its  expected  processing 
demands  per  time  unit.  For  OPT,  APP,  BALANC,  and  RELTOT  the  point  indicates  a 
direction  from  the  origin  which  is  determined  by  ratios  of  intrinsic  demands. 
The  utilization  of  processor  1  is  lower  than  that  of  processor  2,  £al  <  ia2,  such 
that  the  optimal  solution  is  Max  tT\/tx2’  Any  selection  which  is  below  the  line 
tx  i  =  tx 2  is  considered  to  be  better  than  that,  of  a  nonbalancing  scheduler. 
Notice  that  because  BALANC  selects  the  job  which  is  closest  to  the  line 
txz/t* i  =  ^ai/^as.  the  efficiency  of  BALANC  depends  on  the  position  of  this  line 
relative  to  the  candidates. 


Selections  of  macro-schedulers  for  the  case  of  unequal 
total  demand  K  —  \  l,2j,  S'  =  \a],  tal  >  ta2 


Fig.  6. 
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Figure  6  depicts  the  same  system  under  a  different,  load.  Here,  processor  1 
is  more  in  demand  than  processor  2,  a  fact  which  is  reflected  in  their  utiliza¬ 
tions.  In  this  case,  the  performance  of  BALANC  is  close  to  the  optimum,  while 
the  performance  of  RELTOT  has  deteriorated  due  to  the  inequality  in  total 
demand. 


schedulers 

case  1 

>  tash 8 

case  2 

t(L  ih  1  <  tazhz 

case  3 

=  2*b2 

OPT  and  APP 

tx  2 

Max 

tx  1 

Max 

*x2 

any  job  x 

HCLUP 

Max  tx  2 

Max  tx  i 

a ny  job  x 

LCMUP 

Min  tx  i 

Min  tx 2 

any  job  x 

BALANC 

for  all  cases 

^z2  ta\tb  i 

Lz  1  LaZ LbZ 

RELTOT 

_  *0.2 Db  + 

t~2  2  DaDb  4*  t~\tb  1  + 

tx\  ta\ ®b  "t*  tbl-Da 

DaBb  +  ta\tb\  +  ‘a2^b  2 

SRM 

Max  \_tx  l(faS^6  s)  4"  tx2i^a\^b\)  ] 

Table  5.  Selection  criteria  for  two  processors  and  two  active  jobs 


The  selection  criteria  for  the  case  of  two  processors  and  two  jobs  in  the  ser¬ 
vice  set  prior  to  the  selection  of  a  third  active  job  are  summarized  in  Table  5 
(see  Appendix  A  Example  3).  The  results  are  similar  to  the  previous  case,  with 
exceptions  due  to  the  different  expressions  for  utilization: 


Uj(S)  = 


^CLj^b  4" 


for  [1,2]  and  S'  =  [n,b] 


TV, 

X  L  • 


DaDb  -f  taitbi  +  taZtb~ 
ic  last  example  analyzed  is  that  of  three  processors  and  one  active  job 


(see  Example  4).  The  selection  criterion  of  OPT  in  this  case  is  as  follows: 
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tx2  ^x  3 

1  +  + 


Max 


t 


Xi 


t 


xl 


Da  +  lal  +  (Da  +  taS)^  +  (Da  +  <„ 3)-^ 

*x  l  lx  \ 


The  selections  of  APP  and.  OPT  are  identical  here,  but  will  vary  with  an  increased 


multiprogramming  level. 


HCLUP  and  LCMUP  have  good  selection  criteria  but,  again,  with  respect  to 


one  processor  only. 


BALANC  selects  candidate  x  according  to 


^x2  (At”^az) 

- - - — 


and 


>  Wa~ta3) 
^xl  {&a~  tal) 


This  selection  formula  is  not  optimal,  but  BALANC  will  be 


better  than  a  nonbalancing  scheduler.  For  a  comparison  of  BALANC’s  formula 
and  the  optimal  solution  formula  consider  Figures  7a  and  7b.  In  the  first  figure 
the  regions  in  -which  the  target  points  of  BALANC  lie  arc  depicted.  In  the  second 
figure  the  corresponding  directions  of  increased  system  demand  according  to 
the  values  of  ta  j,  taz,  and  <ag  are  delineated.  The  exact  coordinates  of  the  points 
depend  on  processor  utilization.  However,  the  regions  in  which  the  target  points 
fall  match  the  directions  of  increased  system  demand. 


RELTOT’s  selection  in  this  case  is  based  on  the  ratios: 


tzz  ^  (Ta^q  ~  ^2)  .  ^x3  (T3^a  ~  *as) 

*xi  (TtDa  ~  tal)  Qn  txi  (rxDa  ~  tai) 

When  total  demands  are  equal,  rx  -  r2  =  r3,  RELTOT  performs  like  BALANC.  Oth¬ 
erwise,  RET,1 TOT’s  efficiency  is  determined  by  the  values  of  total  processing 
demands. 


The  SRM  algorithm  selects  its  candidate  according  to  the  formula: 

Max  (Da  -  taX)2  txl  +  (JDa  -  ta2)2  tx2+  (Da  -  ta3 f  tx3 

XG  / 

High  consumers  of  underutilized  processors  are  favoured,  so  that  the  perfor¬ 
mance  of  SRM  is  expected  to  be  better  than  that  of  a  nonbalancing  scheduler. 
The  performance  will  fall  short  of  the  optimum,  however,  as  this  formula  strictly 
increases  with  txX,  tx2,  and  tx 3.  Similar  results  are  expected  for  cases  involving 
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^  x2  ^x3 


Fig.  7a.  Target  points  of  the  BALANC  algorithm 


1-  ta1>ta2>ta3 


^  ta2>^a1>ta3 


3  ta1>*a3>^a2 


4  ta2>ta3>ta1 


5-  ta3>ta1>ta2 


6  ta3>ta2>ta1 


f‘ig.  7b.  Corresponding  directions  of  increased  system  demand 


more  than  three  processors. 

From  the  above  examples,  the  following  conclusions  regarding  the  short¬ 
term  objective  are  suggested: 

1.  All  balancing  macro-schedulers  described  so  far  improve,  on  the  average, 
system  imbalance  over  nonbalancing  algorithms. 
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2.  The  performances  of  APP,  BALANC,  RELTOT,  HCLUP,  LCMUP,  and  SRM  are 
not  optimal  and  depend  on  the  characteristics  of  the  inactive  jobs. 

3.  The  efficiency  of  BALANC  increases  with  the  inequality  in  total  demand  for 
processors. 

4.  The  performance  of  RELTOT  equals  that  of  BALANC  when  there  is  an  equal 
total  demand  for  processors.  Otherwise,  RELTOT’ s  efficiency  deteriorates 
with  an  increased  bias  in  the  total  demand. 

5.  In  the  simple  examples  described  above,  the  performance  of  APP  is  the 
same  as  the  performance  of  OPT. 

5.2.  Simulation  Results 

In  order  to  test  the  conclusions  to  this  point,  the  simulation  program 
described  in  Section  3,2  has  been  employed.  The  program  has  been  expanded  to 
allow  for  the  simulation  of  OPT  and  APP.  Also,  job’s  set  vice  demands  which  were 
fixed  prior  to  activation  have  been  made  random  according  lu  the  following  pr  o¬ 
cedure:  every  processor  is  associated  with  a  probability  distribution  function, 
fj.  Upon  arrival,  a  job  is  assigned  a  set  of  values  Cy ,  sampled  from  the  f3-  func¬ 
tions.  When  a  job  is  activated,  the  service  received  is  evaluated  each  time  the 
job  is  assigned  a  processor.  The  amounts  of  service  are  sampled  out  of  an 
exponential  distribution  with  mean  Cy.  With  this  scheme,  Cy  is  the  estimated 
processing  time  received  by  job  i  every  time  processor  j  is  assigned  to  it.  The 
expected  total  processing  demand  of  job  i  from  processor  j  is  MULT-C y,  where 
MULT  is  the  expected  number  of  visits  a  job  makes  to  each  processor. 

Simulation  results  agree  with  the  conclusions  of  the  previous  section  and 
demonstrate  the  behaviour  of  the  macro-schedulers.  In  all  test  runs,  OPT  is  con¬ 
sistently  superior  to  the  other  macro-schedulers.  Second  to  OPT,  the  perfor¬ 
mance  of  APP  appears  to  be  better  than  any  other  balancing  algorithm.  Results 
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of  seven  simulation  runs; are  presented  (Figures  8  to  14). 

In  these  examples,  each  macro-scheduler  was  operating  for  40,000  units  of 
simulated  time.  The  average  of  processor  utilizations  was  calculated  at  regular 
intervals  and  plotted  accordingly.  Utilization  was  measured  over  an  interval 
which  starts  at  the  first  invocation  of  Lhe  scheduler  and  ends  at  the  current 
invocation  time. 

A  parameter  which  is  fixed  for  all  examples  is  MULT  =  10.  Except  for  the 
last  example,  the  micro-scheduling  discipline  is  processor-sharing,  which  is  one 
of  the  acceptable  rules  of  service  in  separable  systems  (see  Section  4.1).  An 
estimated  170  scheduling  intervals  are  included  in  each  of  the  runs  of  Figures  8, 
9,  10,  and  13.  For  Figures  11,  12,  and  14  the  estimated  number  of  scheduling 
intervals  is  87.  Confidence  intervals  were  not  calculated  for  the  results,  since 
these  results  concern  the  short-term  objective  only. 

In  the  first  four  cases,  the  multiprogramming  level  equals  five,  such  that 
OPT  is  based  on  the  last  formula  which  appears  in  Appendix  B.  Parameters  for 
the  first  example  are  an  inactive  set  of  500  jobs,  and  /i  =  -  ft  =  Uni¬ 

form^,  40).  The  latter  are  the  functions  from  which  the  mean  demand  for  a  pro¬ 
cessor  is  sampled.  Results  depicted  in  Figure  3  show  OPT  as  the  most  efficient 
algorithm  improving  average  utilization  over  FCFS  by  19  to  11  percentage 
points.  APP  is  second  best.  Except  for  an  initial  stage  in  which  average  utiliza¬ 
tion  is  more  susceptible  to  random  fluctuations,  the  performance  of  APP  does 
not  fall  below  the  performance  of  OPT  by  more  than  three  percentage  points.  As 
expected  in  a  system  with  a  large  inactive  set  and  with  an  unbiased  demand  for 
processors,  the  efficiencies  of  BALANC  and  RELTOT  are  close,  improving  average 
utilization  by  six  to  nine  percentage  points.  As  well,  the  other  balancing  algo¬ 
rithms  perform  better  than  FCFS.  As  noted  in  Section  3.3,  the  efficiency  of  the 
macro-schedulers  declines  with  time,  because  the  variety  of  candidates 
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decrease?  when  the  inactive  set  is  continuously  subjected  to  one  type  of  selec¬ 
tion. 

Next,  all  parameters  remain  constant  except  for  the  size  of  the  inactive  set 
■which  is  reduced  to  50  jobs  (Figure  9).  In  this  case,  the  macro-schedulers  have  a 
tenfold  smaller  selection  of  candidates  to  choose  from.  As  a  result,  the  improve¬ 
ment  of  the  balancing  algorithms  over  FCFS  is  smaller.  For  instance,  OPT 
increases  average  utilization  by  about  eight  percentage  points  only. 

Results  for  a  system  which  is  identical  to  the  one  considered  in  the  first 
example,  except  for  the  number  of  processors  which  is  increased  to  five,  are 
depicted  in  Figure  10.  Because  the  multiprogramming  level  equals  the  number 
of  processors,  average  utilization  is  quite  low.  Notice  that,  apart  from  OPT  and 
APP  which  are  the  most  efficient,  the  order  of  the  other  balancing  macro¬ 
schedulers  has  changed. 

In  the  following  example  (Figure  11),  there  is  an  unequal  distribution  of 
demands  for  processors:  /3  =  Uniform(0, 10),  /2  =  Uniform(0,30),  and  /3  =  Uni¬ 
form^,  60).  With  an  inactive  set  of  250  jobs,  OPT  and  APP  achieve  the  highest 
productivity.  Initially,  OPT  improves  average  utilization  over  FCFS  by  28  percen¬ 
tage  points.  The  improvement  gradually  decreases  to  about  half  at  the  end  of 
the  run.  APP  trails  OPT  by  at  most  three  points.  As  mentioned  in  the  preceding 
section,  when  total  demands  are  unequal,  the  performance  of  BALANC  improves 
while  the  performance  of  RELTOT  deteriorates  until  it  is  not  much  better  than 
FCFS. 

A  high  multiprogramming  level  is  considered  next.  In  this  case,  OPT  was 
not  implemented  and  only  its  approximation,  APP,  is  examined.  Results  for  a 
system  which  is  identical  to  the  one  in  the  previous  example,  except  for  a  mul¬ 
tiprogramming  level  of  ten  are  illustrated  in  Figure  12.  Here,  APP  is  the  best 
scheduler  with  BALANC  being  a  close  second.  APP  improves  average  utilization 
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over  FCFS  by  27  to  17  percentage  points,  •while  the  efficiency  of  BALANC  is  within 
three  points  from  that  of  APP.  In  Figure  13  the  case  of  expected  equal  total 
demand  for  processors  with  a  multiprogramming  level  of  20  is  depicted.  APP 
remains  the  best,  while  the  performance  of  BALANC  is  relatively  low. 

Finally,  the  macro-schedulers  were  examined  with  a  system  whose  proces¬ 
sors  obey  the  FCFS  micro-scheduling  discipline  (Figure  14).  As  service  is  job 
dependent,  the  FCFS  micro-scheduling  rule  violates  the  conditions  for  a  separ¬ 
able  system.  Even  in  this  case  OPT  is  best,  improving  average  utilization  by  as 
much  as  8.5  points  over  FCFS.  This,  however,  is  way  below  the  performance  of 
OPT  for  the  case  of  processor-sharing  discipline  (Figure  11). 

5.3.  The  Long-Term  Objective 

Results  so  far  indicate  that  the  proposed  balancing  algorithms  improve  sys¬ 
tem  balance  during  scheduling  intervals.  As  mentioned  in  Chapter  One,  achiev¬ 
ing  system  balance  during  scheduling  intervals  is  a  short-term  objective,  and  a 
means  towards  increasing  productivity  over  many  scheduling  intervals. 

The  following  simulation  experiment  was  conducted  to  examine  the  long¬ 
term  effect  of  the  proposed  schedulers.  Given  250  jobs  in  the  inactive  set,  disal¬ 
lowing  arrivals,  all  the  jobs  are  processed  to  completion  under  one  of  the 
schedulers.  Within  one  simulation  run,  different  macro-schedulers  are  tested 
with  identical  workloads.  The  elapsed-time  required  to  process  the  entire  work¬ 
load  is  a  measure  of  the  productivity  achieved. 

The  following  set  of.  parameters  was  chosen  for  the  experiment:  maximum 
multiprogramming  level  of  five,  three  processors,  a  PS  micro-scheduling  discip¬ 
line,  and  MULT  =  10.  Jobs  in  the  system  belong  to  one  of  three  classes.  A  class 
is  distinguished  according  to  the  fj  functions,  from  which  the  means  of  the 
demand  for  processors  are  sampled.  The  following  classes  are  defined: 
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1 .  f  i  =  Uniform  (0,  10),  /2  =  Uniform  (0,  30),  /3  =  Uniform  (0,  60), 

2.  f  i  -  Uniform  (0,  60),  fz  =  Uniform  (0,  10),  /3  =  Uniform  (0,  30), 

3.  / 1  =  Uniform  (0,  30),  fz  —  Uniform  (0,  60),  /3  =  Uniform  (0,  10). 

Class  membership  is  determined  randomly,  with  all  classes  having  equal  proba¬ 

bilities.  Thirty  simulation  runs  were  performed  with  the  same  parameters.  A 
different  seed  has  been,  used  with  each  run.  generating  different  sequences  of 
random  numbers.  The  results  appear  in  Table  6  and  include,  for  each  scheduler, 
the  mean  of  the  eiapsed-time  over  30  runs,  and  the  bounds  of  a  95%  confidence 
interval.  The  algorithms  are  listed  according  to  decreasing  eiapsed-time,  i.e.  in 
increasing  order  of  efficiency.  The  results  indicate  that  the  balancing  algo¬ 
rithms  improve  productivity  over  the  nonbalancing  algorithm.  Improvements 
over  FCFS  range  from  4.4%  under  LCMUP  to  9.7%  under  OPT.  The  improvements 
are  significant,  though  they  are  not  as  substantial  as  the  improvements 
observed  in  some  of  the  experiments  regarding  the  short-term  objective.  The 
vvorkload  in  the  experiment  is  characteristic  of  a  computer  system  in  which 
there  is  a  high  variance  in  the  demands  of  jobs.  Since  simulation  runs  are 
expensive,  a  productivity  study  with  different  workloads  has  not  been  carried 
out. 


scheduler 

iOvVci  uUkalikj. 

elaused-time 

upper  bound 

overhead 

FCFS 

56.7 

57.5 

58.2 

0.0000 

T  r’-UTT'D 

i-J  J.V1  O  A 

54.3 

55.0 

55.7 

0.0003 

KCLUP 

53.8 

54.5 

55.3 

0.0003 

BALANC 

53.5 

54.1 

54.7 

0.0020 

SRM 

53.2 

53.8 

54.5 

0.0017 

RELT0T 

52.3 

53.4 

54.0 

0.0020 

APP 

51.8 

52.3 

52.8 

0.0017 

OPT 

51.3 

51.9 

52.5 

0.0043 

Table  6.  Elapsed-times  required  to  execute  250  jobs. 

The  bounds  are  of  95%  confidence  intervals. 

The  values  are  in  thousands  of  time  units 

The  amount  of  overhead  incurred  by  the  schedulers  during  invocation  times 
is  negligible  in  comparison  to  the  decreases  in  Lhe  eiapsed-time.  Estimated 
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overhead  figures  for  the  last  experiment  are  included  in  Table  6.  The  estimate  is 
based  on  the  following,  factors.  Each  scheduler  is  invoked  240  times.  The 
number  of  operations  per  invocation  time  is  taken  from  Tables  2  and  3.  Since  a 
batch  job  requires  roughly  106  to  109  operations,  we  assume  105  operations  per 
time  unit.  BALANC,  RELTQT,  APP,  and  OPT  require  more  computation  time  dur¬ 
ing  scheduling  intervals.  The  number  of  operations  they  require  per  scheduling 
interval  is  in  the  order  of  hundreds,  such  that  the  overhead  generated  during 
scheduling  intervals  i3  also  negligible. 
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CHAPTER  SIX 


Extensions  and  Conclusions 

The  model  considered  so  far  has  been  restricted  to  separable  networks.  As 
mentioned  in  Section  4.1,  FCFS,  which  is  a  common  micro-scheduling  technique, 
falls  within  the  scope  of  separable  networks  only  if  all  jobs  have  exponential  ser¬ 
vice  time  distributions  with  identical  means.  Reiser  and  Lavenberg  (1978),  and 
Bard  (1979)  suggested  an  approximation  ba.sed  on  mean  value  analysis,  in  which 
the  restriction  of  identical  means  for  the  service  time  distributions,  of  all  classes 
is  relaxed.  Bard  studied  examples  for  which  this  analytical  approximation  pro¬ 
duced  results  which  were  within  4.4%  of  simulation  outcome.  Confidence  inter¬ 
vals,  however,  were  not  included  in  Bard's  study.  The  following  proposed  exten¬ 
sion  is  based  on  the  above  approximation. 

6.1.  An  Extension  of  the  Model  to  Nonseparable  Networks 

Detailed  information  of  job  demands  is  needed  in  order  to  express  system 
demand  as  a  function  of  the  demand  of  a  selected  candidate  under  FCFS  micro¬ 
scheduling  discipline  with  exponential,  class-dependent  service.  Total  service 
time  required  by  job  i  from  processor  j,  ty,  will  be  denoted  as  t ^  where: 

Vij  =  number  of  visits  to  processor  j  required  by  job  i 
cry  =  mean  service  time  per  visit  of  job  i  to  processor  j 
With  this  information,  Reiser  and  Lavenberg  approximate  the  equilibrium  mean 
total  waiting  time  of  job  i  at  processor  j  with  service  set  S'  as: 

S  vajNaj{S-i)\ 

aeS-i 


Mj  )  —  Vy  \_&ij  + 


(9) 
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This  formula  expresses  the  fact  that  under  FCFS,  a  job  which  arrives  at  a  proces¬ 
sor  has  to  wait  for  all  jobs  ahead  of  it  in  the  queue  to  complete  their  service, 
before  it  can  be  served.  Note  that  in  the  special  case  where  a y  is  the  same  for 
all  jobs,  i.e.  when  the  network  is  separable,  Equation  8  reduces  to  Equation  1. 


The  expression  for  the  response  time  of  job  i  after  adding  job  x  to  the  ser 
vice  set  is  as  follows: 


U1)  —  S  Fij(S  {Jx)  -  t  y]  Viy  ^  craj Naj (S  \^Jx  i ) 

i  j  GJC  a  cS"  \Jx  —i 

where  Dt  —  ty  -  the  equilibrium  mean  queue  size  at  a  pro- 

i  *lK  j  <=K 

cessor  with  respect  to  a  single  class  ean  be  computed  according  to  Little  (1961, 
see  also  Equation  2)  as: 


Nij(S)  = 

This  equation,  combined  with  Equation  8,  establish  a  recursive  solution  for  the 
response  time: 


T«(S) 

Ms) 


]C  vij &0.3  Taj (.3  [Jx  i) 

Ri(SlJz)  =  Vf  +  £  *cfr  r  (<,,  ,r-v> - 

acS  \Jx-i  RaiSUx-T') 

The  derivation  of  system  demand,  from  here  on,  is  similar  to  the  the  derivation 
of  system  demand  included  in  Section  4.2. 

For  i  =  x: 


RX(S  \jx)  -  Dx  + 


Yj  njxj  Vaj  Taj(S  ) 

y  j  _ 

iis  k«(s) 


For  i  &  x: 


Ri(S  'Jr)  =  Di  + 


X!  vijCrXjTXj(S  {  jX  i)  y^VijO‘a-jTaj(S  [Jx  i, ) 

S-K _ y  frK  _ 

Rx{S\jx-i)  a^_i  Ra(S\jx-i) 


Substitutions  in  the  second  term  result  in: 
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Ri(S  Ux)  -  Di  + 


I ]VxjVxj[v*}<Txj  +  Vxi  2  <T*sNaj(S-i)] 
jeK  a  eS-i 


Dx  + 


E  vxj^Taj(S-i) 

2  ~ - 


aiS 


R*(S-i) 


(9) 


y~]  °a/  ^  aj  (*b  \j%  0 

yi  _ 

a  h-i  Ra(S'JX-i) 


Combining  with  Equation  3,  system  demand  is  recursively  expressed  as: 


P(S[Jx)  =  [- 


D a 


+  E 


Di 


T,v^ajTaj(S)  £sRi(S{jx) 


■]k 


-1 


Dx  +  a%  Jta(S) 


(10) 


where  Ri{S  ijx)  is  defined  in  Equation  9.  The  analysis  of  particular  examples 
and  the  derivation  of  a  closed-form  expression  are  complicated,  since  the 
number  of  variables  involved  is  double  the  number  of  variables  in  the  solution  of 
separable  networks.  Nevertheless,  a  theorem  is  proven  stating  that  apart  from 
the  mean  service  time  required  by  a  job  per  visit  to  a  processor,  only  the  ratios 
of  the  number  of  visits  a  job  makes  to  each  processor  determine  system 
demand.  The  proof  is  included  in  Appendix  C. 


6.2.  Deletions  from  the.  Service  Set 

So  far,  we  have  addressed  the  problem  of  selecting  a  job  for  addition  to  the 
service  set.  We  have  assumed  that  a  deletion  from  the  service  set  occurs  only 
after  a  job  has  received  its  entire  processing;  In  preemptive  systems,  jobs  are 
allowed  to  be  swapped  out  of  main  storage  prior  to  completion,  i.e.  jobs  can  be 
deleted  from  the  service  set,  waiting  for  reactivation.  The  model  described  in 
Chapter  Four  can  be  extended  to  solve  the  problem  of  deletions  from  the  service 
set.  The  problem  is  to  select  candidate  x  from  service  set  6',  such  that  deleting 
the  job  from  the  set  will  minimize  system  imbalance  over  the  next  scheduling 
interval  and  will  increase  productivity  in  the  long  run.  Under  the  same  assump¬ 
tions  as  in  the  case  of  addition  to  the  service  set,  the  objective  function  is  to 
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maximize  system  demand: 


Max  P(S-x)  = 

z  e  S' 


Substituting  Equations  1  and  4: 


S  tijTCj{S  x  i) 


Ri(S-x)  =  Di  +  y;  tijNj(S -x-i)  =  Di  + 

jzK 


jzK 


c£  S-x-i 


Rc{S-x-i) 


vSimilar  to  the  case  of  addition  to  the  service  set,  a  closed-form  expression  fox- 
system  demand  can  be  generated  given  a  multiprogramming  level. 

We  have  seen  that  the  proposed  balancing  algorithms  produce  improve¬ 
ments  in  system  imbalance  under  various  multiprogramming  levels.  OPT’s  abil¬ 
ity  to  select  jobs  for  deletion  from  and  for  addition  to  the  service  set  implies 
that,  with  OPT,  high  productivity  can  be  achieved  under  a  varying  multiprogram¬ 
ming  level. 

8.3.  Conclusion 

Macro-scheduling  algorithms  for  increasing  system  productivity  have  been 
investigated.  Systems  with  a  variety  of  processors  were  considered,  such  that  in 
order  to  increase  productivity,  an  attempt  was  made  at  each  invocation  time  to 
balance  the  system,  i.e.  to  bring  the  utilization  of  processors  to  prespecified  lev¬ 
els.  To  simplify  the  presentation,  processors  were  considered  to  be  equally 
important,  and  the  prespecified  levels  of  utilization  were  set  at  1.0.  As  a  result, 
the  short-term  objective  was  to  maximize  the  average  of  processor  utilizations. 
Comparative  analytic  studies  and  simulation  studies  of  seven  balancing  macro¬ 
schedulers  demonstrated  that  significant  gains  in  productivity  can  be  a.chieved 
by  employing  such  algorithms.  It  has  been  shown  how  the  proposed  macro¬ 
schedulers  can  be  applied  to  processors  of  different  importance,  to  target  bal¬ 
ance  values  which  are  less  than  1.0,  and  to  a  subset  of  the  processors.  Addi¬ 
tional  experiments  will  be  needed  to  determine  what  gains  can  be  achieved  by 


-  74  - 


employing  balancing  schedulers  in  particular  situations.  However,  since  Lne 
ranking  of  the  schedulers  in  the  short-term  experiments  matches  the  ranking  in 
the  long-term  experiment,  we  can  assume  that  a  scheduler  which  achieves 
better  utilization  control  during  scheduling  intervals  will  produce  more  gains  in 
productivity  over  long  intervals.  The  ranking  of  the  algorithms  in  cases  involving 
processors  of  different  importance,  target  balance  values  which  are  less  than 
1.0,  and  a  subset  of  the  processors  is  expected  to  be  similar  to  the  ranking 
obtained  in  the  experiments  involving  the  short-term  objective  (see  Chapter 
Five). 

Initially,  algorithms  have  been  designed  to  implement  traditional 
approaches  for  increasing  system  productivity,  such  as  loading  the  highest  con¬ 
sumer  of  an  underutiiized  processor,  or  loading  the  job  which  is  expected  to 
contribute  the  most  to  equalize  processor  utilizations.  In  some  simulations 
these  algorithms  increased  the  average  of  processor  utilizations  by  as  much  as 
15  pcccntagc  points  over  a  neutral  discipline.  However,  none  of  the  examined 
schedulers  seemed  to  be  superior  to  the  others,  as  statistical  variations  in  the 
workload  caused  a  change  in  the  ranking  of  the  algorithms  according  to  the 
average  utilization  achieved. 


In  order  to  enhance  our  understanding  of  productivity  algorithms,  a  system 
model  has  been  employed.  The  use  of  a  model  was  necessary  to  correlate  the 
choice  of  the  candidate  with  the  resulting  system  imbalance.  To  this  end,  a 
separable,  Markovian  queueing  network  model  of  the  system  has  been 
developed.  Within  the  framework  of  the  model,  average  utilization  was 
expressed  as  a  recursive  function  of  the  intrinsic  demand  ratios  of  the  selected 


job.  One  result  of  this  analysis  was  the  insight  gained  into  the  dependence  of 
algorithm  performance  on  the  make-up  of  the  inactive  set,  and  on  the  utilization 
of  processors.  Another  outcome  of  the  analysis  was  the  development  of  OPT,  a 
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macro-scheduling  algorithm  which  is  optimal  with  respect  to  the  model  assump¬ 
tions.  Simulation  studies  indicated  that  OPT  performed  better  than  the  other 
macro-schedulers,  even  when  assumptions  of:  the  model  were  violated.  In  some 
examples,  improvements  in  average  processor  utilization  of  up  to  28  percentage 
points  over  FCFS  scheduling  were  observed. 

A  drawback  of  OPT  might  be  its  computational  complexity  which  is  propor¬ 
tional  to  k  s\,  where  k  is  the  number  of  processors  and  s  is  the  multiprogram¬ 
ming  level.  Much  of  the  calculation  however  can  be  carried  out  before  invoca¬ 
tion  time,  such  that  only  4fc+3  arithmetic  operations  are  required  to  evaluate  a 
candidate.  For  the  case  of  a  large  service  set,  an  approximate  algorithm,  APP, 
which  is  computationally  linear  in  s  has  been  suggested.  In  a  simulation  study 
involving  a  small  number  of  processors,  the  performance  of  APP  is  within  three 
percentage  points  of  the  performance  of  OPT,  and  is  better  than  the  perfor¬ 
mance  of  the  other  algorithms. 

Other  approximations  of  OPT  can  be  based  on  approximate  solution 
methods  for  large  separable  queueing  networks.  The  method  of  class  aggrega¬ 
tion  (Zahorjan  19B0)  seems  to  be  particularly  suitable  for  the  case  of  a  large  ser¬ 
vice  set.  Tradeoff  between  the  accuracy  gained  and  the  overhead  introduced  by 
new  approximations  should  be  investigated. 

The  long-term  effect  of  the  proposed  schedulers  was  examined,  under  a  high 
variance  workload.  95%  confidence  intervals  were  established  for  the  results.  In 
this  typical  example,  all  balancing  algorithms  increased  productivity  compared 
to  a  neutral  algorithm.  OPT  produced  the  highest  productivity.  APP  was  second 
to  OPT.  Elapsed-times  required  to  execute  a  given  workload  were,  on  the  aver¬ 
age,  9.7%  shorter  under  OPT  than  under  FCFS  scheduling.  Although  the  long- 
term  effect  of  the  macro-schedulers  has  not  been  examined  with  different  types 
of  workloads,  it  is  obvious  that  a  high  variance  workload  is  critical  for  achieving 


-  76  - 


substantial  gains  in  productivity.  Results  of  the  short-term  experiments  indi¬ 
cate  that  the  balancing  algorithms  become  less  effective  with  a  smaller  variance 
workload.  In  the  extreme  case  of  zero  variance,  productivity  is  independent  of 
the  macro-scheduling  discipline. 

The  choice  of  a  balance  point  leading  to  maximum  productivity  is  an  open 
question.  Two  arguments  suggest  that  the  point  of  system  balance  should  be 
chosen  according  to  maximum  utilization  of  prooessors.  The  first  argument  per¬ 
tains  to  the  results  of  the  long-term  experiment.  The  results  indicate  that  algo¬ 
rithms  which  achieve  higher  levels  of  utilization  in  the  short  run  provide  higher 
productivity  in  the  long  run.  The  second  argument  is  that  if  the  coordinates  of 
the  balance  point  represent  a  set  of  processors  of  equal  weight  (importance),  an 
optimum  is  achieved  ■when  the  values  of  processor  utilizations  are  equal  (see 
references  in  Section  2.2).  Nonetheless,  for  the  following  reasons,  it  is  often 
desirable  to  maintain  utilizations  at  levels  which  are  less  than  the  maximum 
possible:  1.  the  necessity  to  keep  important  processors  highly  utilized,  2.  the 
necessity  to  accomodate  user-oriented  objectives,  and  3.  the  necessity  to  avoid 
high  overhead.  In  systems  where  the  CPU  is  the  most  important  (expensive)  pro¬ 
cessor,  high  CPU  utilization  is  often  achieved  by  keeping  low  utilization  at  other 
processors.  In  HCLUP,  LCMUP,  BALANC,  RELTOT,  and  SRM  the  value  of  system 
balance  can  be  adjusted  according  to  processor  importance.  In  OPT  the  impor¬ 
tance  of  processors  is  taken  into  consideration  in  the  formula  for  system  bal¬ 
ance  (see  Section  4.4-),  such  that  the  value  of  system  balance  need  not  be 
adjusted. 

System  overhead  is  another  factor  which  determines  utilization. 
Remember  that  processor  utilization  has  been  defined  with  respect  to  service 
provided  to  active  jobs,  excluding  overhead.  In  reaiity,  utilization  of  i.O  cannot 
be  achieved  due  to  the  overhead  required  for  micro-scheduling,  software  moni- 
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toring,  and  other  system  tasks.  Overhead  often  increases  with  higher  utilization. 
In  a  system  with  a  virtual  memory,  for  example,  many  jobs  might  have  to  be 
activated  to  ensure  high  utilization.  An  increase  in  the  number  of  active  jobs 
will  cause  an  increase  in  overhead  due  to  higher  activity  of  the  paging  system. 
In  HCLUP,  LCMUP,  BALANC,  RELTOT,  and  SRM  the  point  of  system  balance  should 
be  set  such  that  excessive  overhead  is  avoided.  In  the  queueing  network  model 
discussed  in  Chapter  Four,  overhead  can  be  incorporated  into  the  model.  Une 
approach  is  to  add  extra  active  jobs  which  belong  to  a  new  class.  The  demands 
of  the  extra  jobs  represent  the  overhead.  Another  approach  to  representing 
overhead  in  queueing  network  models  is  to  inflate  the  demands  of  jobs  to 
account  for  overhead  introduced  (Kritziriger,  Krzesinski  and  Teunissen  1930). 

The  last  reason  mentioned  for  having  processors  utilized  at  levels  lower 
than  the  maximum  is  to:  ensure  adequate  response  to  users.  However,  the  value 
of  system  balance  need  not  be  adjusted  to  accomodate  user-oriented  objectives. 
The  proposed  productivity  schedulers  will  be  invoked  when  user-oriented  objec¬ 
tives  are  secondary  to  the  objective  of  high  productivity,  or  when  another 
scheduler  pursues  the  user-oriented  objectives. 

A  productivity  scheduler  can  be  the  only  macro-scheduler  in  the  system,  in 
cases  where  user  objectives  are  insignificant  compared  to  the  objective  of  high 
productivity.  A  productivity  scheduler  can  also  be  just  part  of  the  macro- 
scheduling  process.  As  mentioned  in  Chapter  One,  a  productivity  scheduler  can 
either  be  employed  continuously,  in  parallel  with  other  macro-schedulers  which 
pursue  different  objectives,  or  occasionally,  when  special  conditions  arise.  In 
the  former  case,  the  productivity  scheduler  will  not  select  a  candidate  for  swap¬ 
ping,  but  will  evaluate  all  candidates  and  assign  a  recommended  value,  Vs,  to 
each  one  according  to  its  expected  contribution  towards  system  balance.  At  the 
same  time,  a  different  algorithm  will  produce  a  set  of  user-oriented  recom- 
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mended  values,  Vu.  Some  combination  of  these  recommendations,  e.g.  F.,+CVST 
will  be  the  criterion  for  job  selection,  where  C  is  a  weighting  factor.  C  may  be 
changed  according  to  system  load  and  processor  imbalance. 

A  macro-scheduler  can  have  two  scheduling  modules,  each  designed  to  pur¬ 
sue  a  different  objective,  but  only  one  active  at  each  invocation  time.  Normally, 
the  user-oriented  scheduler  will  be  active,  but  once  a  certain  level  of  load  or 
level  of  imbalance  is  reached,  the  productivity  scheduler  will  be  employed  for  a 
short  while  until  the  situation  is  alleviated.  A  productivity  macro-scheduler 
could  be  invoked  alone  to  select  a  job  for  swapping  with  high  system  load  or  with 
unbalanced  processors.  Following  such  situations,  the  operating  system  will 
switch  back  to  activate  the  user-oriented  scheduler. 

The  subject  of  combining  the  recommendations  of  scheduling  modules  with 
different  orientations  which  operate  simultaneously,  or  determining  the  thres¬ 
holds  for  alternating  between  schedulers  requires  additional  research. 

Further  research  is  needed  to  extend  the  proposed  separable  queueing  net¬ 
work  model.  The  assumption  of  load  independent  sendee  can  be  limiting  in  some 
applications,  especially  for  modelling  disks  and  channels.  The  extent  of  load 
dependency  is  a  function  of  device  characteristics  and  of  micro-scheduling  dis¬ 
cipline.  There  is  a  tradeoff  between  the  accuracy  of  the  model  and  the  complex¬ 
ity  of  the  solution.  The  degree  of  accuracy  required  in  the  model  should  be 
enough  to  achieve  a  rating  of  the  candidates.  "Whether  the  required  accuracy  can 
be  achieved  under  the  assumption  of  load  independence  should  be  tested  in 
specific  cases.  There  are  solutions  of  queueing  networks  with  load  dependent 
devices,  including  mean  value  analysis  (Reiser  and  Lavenberg  1980).  However, 
the  solution  of  models  with  load  dependent  devices  is  much  more  complex  than 
with  load  independent  devices,  since  marginal  queue  length  probabilities  need  to 


be  calculated. 
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"Within  separable  queueing  network  models,  it  is  impossible  to  represent 
accurately  a  situation  whereby  jobs  wait  in  one  queue  for  service  at  a  number,  of 
devices.  Therefore,  a  channel  and  the  disks  connected  to  it  cannot  be  both  expli¬ 
citly  included  in  a  separable  queueing  modeL  In  performance  analysis  either 
the  channel  or  the  disks  are  represented  as  devices.  In  cases  where  utilization  is 
to  be  controlled  over  the  channel  and  over  the  disks  connected  to  the  channel, 
other  models  should  be  investigated. 

Vital  for  the  successful  implementation  of  productivity  macro-schedulers 
are  predictions  of  jobs  processing :  demands  from  user’s  supplied  information, 
monitored  data,  or  a  combination  of  both.  In  some  production  systems,  predic¬ 
tions  based  on  monitored  data  are  used  for  macro-scheduling  (IBM  1370).  Some 
work  has  been  published  on  this  problem,  e.g.  by  Badel  and  Leroudier  (1978), 
but  more  needs  to  be  done.  This  thesis  provides  the  incentive  for  additional 
research  into  such  predictors,  since  significant  gains  in  productivity  can  be 
achieved  when  good  predictors  are  available. 


1.  Badel,  M.,  Gelenbe,  E..  Leroudier.  J.,  and  Potier.  D.  (1975).  Adaptive  optimiza¬ 
tion  of  a  time-sharing  system’s  performance.  Proc.  IEEE  63,  6  (June 
1975). 


2.  Badel,  M.,  and  Leroudier,  J.  (1978). 


and.  iiiiplemeiitatiuii 


Oii.  IRI 


uA 


Laboria, 


Optimal  multiprogramming:  principles 
Le  Chesnay,  Prance.  Rapport  276,  Jan. 


1978. 


*3  Cl.  ^  C*  r>  r>  ^oVnurof  rn  a  n 

closed  network  models  -  a  solution 
Toronto,  Aug.  1977,  559-564. 


K.D.  (1977).  Customer  classes  and 
tenhninue.  Proc.  of  1FIP  Congress. 

x.  ._> 


4.  Bard,  Y.  (1979).  Some  extensions  to  multiclass  queueing  network  analysis. 
4th  Int’l  Symp.  on  Modelling  and  Performance  Evaluation  of  Computer  Sys¬ 
tems,  Vienna,  Feb.  1979. 


5.  Baskell,  F.,  Chandy,  K.M.,  Muntz,  R.R.,  and  Palacios,  F.G.  (1975).  Open,  closed, 

and  mixed  networks  of  queues  -with  different  classes  of  customers.  J.  ACM 
33,  2  (Apr.  1975),  248-260. 

6.  Belady,  L.A..  and  Kuehner,  C.J.  (1969).  Dynamic  space-sharing  in  computer 

systems.  Comm.  ACM  12,  5  (May  1969),  282-288. 

7.  Bernstein,  A.J.,  and  Sharp,  J.C.  (1971).  A  Policy-driven  scheduler  for  a  time¬ 

sharing  system.  Cornm.  ACM  14,  2  (Feb.  1971),  74-78. 

8.  Brinch-Hansen,  P.  (1971).  An  Analysis  of  response  ratio  scheduling.  Proc.  of 

IF1P  Congress,  Ljubljana,  Aug.  1971,  480-484. 

9.  Brinch-Hansen,  P.  (1973).  Operating  systems  principles.  Prentice-Kali,  Engle¬ 

wood  Cliffs,  N.J.,  1973. 


10.  Bunt,  R.B.  (1974).  Self-regulating  schedulers  for  operating  systems.  Ph.D. 

Thesis,  University  of  Toronto,  1974. 

11.  Bunt,  R.B.,  and  Hume,  J.N.P.  (1972).  Self-regulating  operating  systems. 

INFOR  10,  3  (Oct.  1972),  232-239. 

12.  Buzen,  J.P.  (1971).  Analysis  of  system  bottlenecks  using  a  queueing  network 

model.  Proc.  ACM  SIGOPS  Workshop  on  System  Performance  Evaluation, 
Harvard  University,  Cambridge,  Mass.,  April  1971,  82-103. 

13.  Buzen,  J.P.  (1973).  Computational  algorithms  for  closed  queueing  networks 

with  exponential  servers.  Comm.  ACM  16,9  (Sept.  1973),  527-531. 


-  81  - 


14.  Buzen,  J.P.  (1978).  A  Queueing  network  model  of  MVS.  Computing  Surveys 

10,  3  (Sept.  1978),  319-331. 

15.  Buzen,  J.P.  (1980).  Private  communication. 

18.  Chandy,  K.M.,  Herzog,  U.,  and  Woo,  L.  (1975).  Parametric  analysis  of  queuing 
networks.  IBM  J.  Res.  Develop .  19,  1  (Jan.  1975),  36-42. 

17.  Chua,  Y.S.,  and  Bernstein,  A.J.  (1974).  Analysis  of  a  feedback  scheduler. 

SIAM  J.  Comput.  3,  3  (Sep.  1974),  159-176. 

18.  Coffman,  E.G.Jr.,  and  Kleinrock,  L.  (1968).  Feedback  queueing  models  for 

time-shared  systems.  JACM  15,  4  (Oct.  19BR),  549-576. 

19.  Coffman,  E.G.Jr.,  and  Mitrani,  I.  (1975).  Selecting  a  scheduling  rule  that 

meets  pre-specified  response  time  demands.  Proc.  of  5th  Symp.  on 
Operating  Systems  Principles,  SIGOPS,  Austin,  Texas,  Nov.  1975,  187-191. 

20.  Conway,  R.W.,  Maxwell,  W.L.,  and  Miller,  L.W.  (1967).  Theory  of  scheduling . 

Addison-Weslejr,  1967. 

21.  Courtois,  P.J.  (1977).  Decomposability;  queueing  and  computer  sijstem 

applications .  ACM  monograph  series,  Academic  Press,  Inc.,  1977. 

22.  Denning,  P.J.  (1968a).  Resource  allocation  in  multiprocess  computer  sys¬ 

tems.  Ph.D.  Thesis,  M.I.T.,  Project  MAC,  MAC-TR-50,  1968. 

23.  Denning,  P.J.  (1968b).  The  Working  set  model  for  program  behaviour.  Comm. 

ACM  11,  (1968),  323-333. 


24.  Denning,  P.J.  (1969).  Equipment  configuration  in  balanced  computer  sys¬ 

tems.  IEEE  Trans.  Computers  C-1 8,  11  (Nov.  1969),  1008-1012. 

25.  Denning,  P.J.  (1971).  Third  generation  computer  systems.  Computing  Sur¬ 

veys  3,  4  (Dec.  1971),  175-216. 

26.  Denning,  P.J.,  Kahn,  K.C.,  Leroudier,  J.,  Potier,  D.,  and  Suri,  R.  (1976). 

Optimal  multiprogramming.  Acta  Informa tica  7,  2  (1976),  197-216. 

27.  Geek,  A.  (1979).  Performance  improvement  by  feedback  control  of  the 

operating  system.  4th  Int’l  Symp.  on  Modelling  and  Performance  Evalua¬ 
tion  of  Computer  Systems,  Vienna,  Feb.  1979. 

28.  Ghanem,  S.B.  (1975).  Computing  center  optimization  by  a  pricing-priority 

policy.  IBM  SysL  J.  14,  3  (1975),  272-291. 

29.  Gordon,  W.T.,  and  Newell,  G.F.  (1967).  Closed  queueing  systems  with 

exponential  servers.  Operations  Research  15,  April  1967,  252-265. 

30.  Graham,  G.S.  (Ed.).  (1978).  Special  issue:  queueing  network  models  of  com¬ 

puter  system  performance.  Computing  Surveys  10,  3  (Sept.  1978). 

31.  Greenberger,  M.  (1966).  The  Priority  problem  and  computer  time  sharing. 

Management  Science  12,  11  (July  1966),  888-906. 


-  02  - 


32.  IBM  0S/VS2  MVS.  System  programming  library:  initialization  and  tuning 

guide.  Part  5:  the  system  resources  manager.  GC28-0681-3,  May  1978, 
169-237. 

33.  Jackson,  J.R.  (1963).  Jobshop-like  queueing  systems.  Management  Science 

10,  1  (Oct.  1963),  131-142. 

34.  Kameda,  H.  (1975).  The  Analysis  of  an  adaptive  workload  balancing  strategy 

in  computing  system  resources  management.  International  Journal  of 
Computer  and  Information  Sciences  4,  4  (1975),  295-306. 

35.  Kameda,  H.,  and  Gotlieb,  C.C.  (1977).  A  Feedback-coupled  resource  alloca¬ 

tion  policy  for  multiprog  rammed  computer  systems.  Acta  Informatica  8, 
4  (1977),  341-357. 

36.  Kienzle,  M.G.,  and  Sevcik,  K.C.  (1979).  Survey  of  analytic  queueing  network 

models  of  computer  systems.  Conference  on  Simulation,  Measurement 
and  Modeling  of  Computer  Systems,  Boulder,  Colorado,  Aug,  1979,  113-129. 

37.  Kim,  K.H.,  and  Naghibzadeh,  M.  (1980).  Prevention  of  task  overruns  in  real¬ 

time  non-preemptive  multiprogramming  systems.  Proc.  of  Performance 
80,  7th  Int’l  Symp.  on  Computer  Performance  Modelling,  Measurement  and 
Evaluation,  IFIP  W.G.  7.3,  May  1980. 

38.  Kieinrock,  L.  (1970).  A  Continuum  of  time-sharing  scheduling  algorithms. 

Proc.  AFIPS  1970  SJCC,  Vol.  36,  453-458. 

39.  Kieinrock,  L.  (1976).  Queueing  systems ,  volume  II:  computer  applications. 

John  Wiley  and  Sons,  New  York,  1976. 

40.  Kritzinger,  P.S.,  Krzesinski,  A.E.,  and  Teunissen,  P.  (1980).  Incorporating  sys¬ 

tem  overhead  in  queuing  network  models.  IEEE  Transactions  on  Software 
Engineering  SE-6,  4  (July  1980),  381-390. 

41.  Lawler,  E.L.  (1976).  Combinatorial  optimization:  networks  and  matroids. 

Holt,  Rinehart  and  Winston,  New  York,  1976. 

42.  Leroudier,  J.,  and  Potier,  D.  (1976).  Principles  of  optimality  for  multipro¬ 

gramming.  Proc.  Tnt.'l  Symp.  on  Computer  Performance  Modeling,  Meas¬ 
urement  and  Evaluation,  ACM-SIGMETRICS  and  IFIP  W.G.  7.3,  Cambridge, 
Mass.,  March  1976,  211-218. 

43.  Levin,  R.,  Cohen,  E.,  Corwin,  W.,  Pollack,  F.,  and  Wulf,  W.  (1975). 

Policy/mechanism  separation  in  HYDRA.  Proc.  of  5th  Symp.  on  Operating 
Systems  Princinples,  SIG0PS,  Austin,  Texas,  Nov.  1975,  132-140. 

44.  Little,  J.D.C.  (1961).  A  Proof  for  the  queueing  formula  L  =  XW.  Operations 

Research  9,  3  (May  1961),  383-387. 

45.  Lynch.  H.W.,  and  Page,  J.B.  (1974).  The  0S/VS2  release  2  system  resources 

manager.  IBM  Syst.  J.  13,  4  (1974),  274-291. 

46.  Mahl,  R.  (1970).  An  Analvtic  approach  to  computer  systems  scheduling. 

Ph.D.  Thesis,  University  of  Utah,  UTECH-CSc-70-100,  1970. 


-  83  - 


47.  McKinney,  J.M.  (1989).  A  Survey  of  analytical  time-sharing  models.  Comput¬ 

ing  Surveys  1 ,  2  (June  1969),  105-116. 

48.  Michel,  J.A.,  and  Coffman,  E.G.Jr.  (1974).  Synthesis  of  a  feedback  queueing 

discipline  for  computer  operation.  J.  ACM  21 ,  2  (April  1974),  329-339. 

49.  Mitrani,  I.,  and  Kine,  J.H.  (1977).  Complete  parametrized  families  of  job 

scheduling  strategies.  AclalnformaticaS ,  1  (1977),  61-73. 

50.  Performance  80.  (1980).  Proc.  of  Performance  80,  7th  Int’l  Symp.  on  Com¬ 

puter  Performance  Modelling,  Measurement  and  Evaluation,  IFIP  W.G.  7.3, 
May  1980. 

51.  Reiser,  M.,  and  Kobayashi,  H.  (1975).  Queueing  networks  with  multiple  closed 

chains:  theory  and  computational  algorithms.  IBM  J.  Res.  and  Develop.  19, 
3  (May  1975),  283-294. 

52.  Reiser,  M.  and  Lavenberg,  S.S.  (1978).  Mean  value  analysis  of  closed  mul¬ 

tichain  queueing  networks.  IBM  Research  Report  RC  7023,  IBM  T.J.  Watson 
Research  Center,  Yorktown  Heights,  New  York,  Oct.  1978. 

53.  Reiser,  M.,  and  Lavenberg,  S.S.  (1980).  Mean-value  analysis  of  closed  mul¬ 

tichain  queueing  networks.  J.  ACM  27,  2  (April  1980),  313-322. 

54.  Reiter,  A.  (1967).  A  Resource-allocation  scheme  for  multi-user  on-line  opera¬ 

tion  of  a  small  computer.  Conf.  Proc.  AFIPS  SJCC  30,  Atalantic  City,  N.J., 
(Apr.  1967). 

55.  Roscnfeld,  J.L.  (1977).  SRM  load  balancers.  Unpublished  paper.  IBM  T.J.  Wat¬ 

son  Research  Center,  Yorktown  Heights,  New-York,  1977. 

56.  Rothkopf,  M.H.  (1966).  Scheduling  with  random  service  times.  Management 

Science  12,  9  (May  1966),  707-713. 

57.  Ruschitzka,  M.  (1978).  An  Analytical  treatment  of  policy  function  schedulers. 

Operations  Research  26,  5  (Sept. -Oct.  1978),  845-863. 

58.  Ruschitzka,  M.  (1979).  A  Load-sensitive  scheduler  for  interactive  systems. 

4th  Int’l  Symp.  on  Modelling  and  Performance  Evaluation  of  Computer  Sys¬ 
tems,  Vienna,  Feb.  1979. 

59.  Ruschitzka,  M.  (1980).  The  Response  of  job  classes  with  distinct  policy  func¬ 

tions.  Proc.  of  Performance  80,  7th  Int'l  Symp.  on  Computer  Performance 
Modelling,  Measurement  and  Evaluation,  IFIP  W.G.  7.3,  Toronto,  May  1980. 

60.  Ruschitzka,  M.,  and  Fabry,  R.S.  (1977).  A  Unifying  approach  to  scheduling. 

Comm.  ACM  20,  7  (July  1977),  489-476. 

61.  Scherr,  A.  (1967).  An  Analysis  of  a  time-shared  computer  system.  MIT  Press, 

Cambridge,  Mass.,  1967. 

62.  Schrage,  L.  (1968).  A  Proof  of  the  optimality  of  the  shortest  remaining  pro¬ 

cessing  time  dicsipline.  Operations  Research  16,  3  (May-June  1968),  687- 
690. 


-  84  - 


63.  Serlin,  0.  (1972).  Scheduling  of  time  critical  processes.  Proc.  AFIPS  1972 

3JCC,  Vol.  40,  925-932. 

64.  Sevcik,  K.C.  (1974).  Scheduling  for  minimum  total  loss  using  service  time 

distributions.  J.  ACM  21,  1  (Jan.  1974),  66-75. 

65.  Sevcik,  K.C.,  and  Mitrani,  I.  (1979).  The  Distribution  of  queueing  network 

slates  al  input  and  output  instants.  4th  Int’i  Syrnp.  on  Modeling  and  Per¬ 
formance  Evaluation  of  Computer  Systems,  Vienna,  Feb.  1979. 

66.  Streeter,  D.N.  (1972).  Cost-benefit  evaluation  of  scientific  computing  ser¬ 

vices.  IBM  Syst  J.  11,3  (1972),  219-233. 

67.  Wallace,  V.L.,  and  Mason,  D.L.  (1969).  Degree  of  multiprogramming  in  page- 

on-dernand  systems.  Cvrrvrri.  ACM  12,  6  (June  1969),  305-308. 

68.  Wulf,  W.A.  (1969).  Performance  monitors  for  multi-programming  systems. 

Proc.  2nd  ACM  Symp.  on  Operating  Systems  Principles,  Oct.  1969,  175-181. 

69.  Zahorjan,  J.  (1980).  The  Approximate  solution  of  large  queueing  network 

models.  Ph.D.  Thesis,  University  of  Toronto,  Technical  Report  C5RG-122, 
Aug.  1930. 


-  85  - 


APPENDIX  A 

Four  examples  of  formula  7  are  presented.  The  examples  differ  in  the 
number  of  processors,  k  -  \K\,  and  in  the  multiprogramming  level  at  invocation 
time,  s  =  j  5  | . 


Example  1.  K  -  2)  ,  S  =  p  (empty  set) 

p(su«>  =  £-7£-  =  °-<» 

Any  job  can  be  loaded  because,  regardless  of  its  demands,  system  demand  will 
be  0.5.  The  reason  is  that  at  each  instant,  with  only  a  single  active  job,  one  pro¬ 
cessor  is  busy  and  one  idle. 

Example  2.  K  -  [\,2\  ,  S  —  \a] 


P{S\Jx)  = 


Df 


Rx(a\Jx)  Ra(a\Jx) 


.-1 


Da 


Dx  +  tx iN i(a)  +  tx2N2(a)  Da  +  £  tajTxj(x)/Rx(x) 

i*K 


.-1 


Substituting  N  ^a)  =  Tal(a)/Ra(a)  =  ta^/(tal+ta2)  =  tal/Da,  Ar2(a)  =  ta2/Da, 
Txj(x)  =  txj>  and  Rxix)  ~  Dx' 


P(S{jx)  = 


D, 


Dr 


l.-  1 


Dx  14  l/^a  ^xZ^aZ/^D  a  Da  +  ta\txi/'Dx  4-  ^aZ^xZ /Dx 

dividing  the  numerator  and  the  denominator  by  in  order  to  decrease  the 

number  of  unknowns,  and  denoting  tx2/tz\  by  dz2: 


P{S  [Jx)  =  Da  • 


1  +  <St2 


Da  +  f-a\  +  {Da  +  taz)fixZ 

The  function  P(S  (Jx)  is  plotted  in  Figure  15.  Three  eases  are  distinguished  in 


the  region  of  interest,  <5*g^0: 


1-  4i  >  t-aZ- 


Max  P(S  \jx)  =  Da/{Da  +  ta2)  can  be  achieved  by  <5*2 
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P(S\jx) 


Fig.  15. 


_ 1  +  $x2 _ 

■Da  ta\  +  (Da  +  ^a2)^z2 


approaching  infinity. 

2.  tal  <  ta2.  Max  P(S  {Jx)  =  Da/(Da  +  tai)  can  be  achieved  by  dx2  approach¬ 
ing  zero. 

3.  i  =  *tx2-  P(S  U1)  =  Da/{Da+ta2)  =  2/3  regardless  of  x. 


The  macro-scheduler  in  this  example  will  perform  as  follows:  if  £al  >  ta2,  the  can¬ 
didate  with  the  largest  ratio  tTz/tx  i  will  be  recommended  for  loading.  If 
tai  <  ta2,  the  candidate  with  the  smallest  such  ratio  will  be  recommended.  Oth¬ 
erwise,  any  candidate  can  be  loaded.  Note  that  when  dx2  =  0  and  £al  —  0,  or 
when  ox2  approaches  infinity  and  ta 2  =  0,  highest  system  demand  is  attained. 
This  is  obvious  since  either  £al  =  tx2  =  0  or  ta2  =  txi  =  0,  but  not  both,  means 
that  each  job  requires  a  different  processor,  such  that  the  two  processors  are 
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fully  utilized. 


Example  3.  K  -[1,2]  ,  <5*  -  {a, b\ 


P{S\Jx)  = 


Da 


D, 


D> 


i--1 


Rx(S\Jx)  Ra{S\jx)  Rb(S  'Jz) 

Employing  the  recursive  formulae  1  and  4-6,  dividing  the  numerator  and  the 

denominator  by  txl,  and  denoting  8xZ  —  txz/txi  results  in: 


P{S  u*)  = 


2DaDb  +  faiffei  +  (2  DaDb  + 


DaDb  +  2ta\tbi  +  (DaDb  4-  2ta2tb2)^x2 
Similar  to  Example  2,  the  following  cases  are  distinguished: 

2DaDb 


1_ 

2 


1.  *ai4i>w&2-  Max  P{S  \Jx)  = 


6Z 2  approaching  infinity. 


2-  ta\tbi  <  ta2th2.  Max  P(S  {jx)  = 


Da Db  2^2^62 


ZDgD  b^tg\tb  1 

DaDb  +  2ta\tb  i 


•  —  can  be  achieved  by 
2  7 


can  be  achieved  by 


6X2  equals  zero. 

3.  =  fa2^>z-  The  value  of  P(S  {Jx)  does  not  depend  on  x. 

In  other  words,  if  processor  1  is  more  utilized  than  2,  candidate  x  with  the 
highest  ratio  £X2/%i  will  be  recommended  for  loading.  If  the  reverse  is  true, 
then  the  candidate  is  chosen  such  that  tTz/tx\  is  minimum.  If  both  processors 
are  equally  utilized,  any  job  can  be  loaded. 

Example  4.  K  -  { 1,  2,  3$  ,  &  =  \a\ 


/  o  •  I  N 
(O 


Da 


Da 


Rx(a[jx)  Ra(a\jx) 

Dr. 


i.-l 


D 


a 


DT  +  E  IxjNjia)  Da  +  Yj  LaiTxj(x)/Rx(x) 
0*K  j*K 


k 


-l 


Substitutions  result  in: 


R{R[JX)  -  q  Da 


1  +  (5^2  +  5X2 


Da  +  1  +  ( Da  +  fa2)fl*8  +  ( Da  +  ta3)6x3 
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where  6xZ  =  txZ/tx i  &X3  =  tx3/tx^.  The  function  P(S'jx)  is  delineated  in  Figure 
16  according  to  two  of  the  six:  possible  relations  among  the  values  of  £al,  ta 2,  and 

t*3’. 

1.  tai>  taZ  >  fa3.  In  the  positive  quadrant,  the  value  of  P(S{Jx)  is  lowest  over 

the  origin.  It  increases  faster  along  than  along  6xZ,  to  an  upper  bound  of 
2Da/3(Da+taZ).  An  inflection  line  exists  in  ^3  =  such  that  if 

<5x3  is  below  this  value,  P(S{Jx)  increases  with  6xZ,  otherwise  it  decreases.  In 
order  to  select  a  job  which  maximizes  P(S{Jx),  the  demands  of  each  candidate 
have  to  be  substituted  in  the  formula  for  P(S  \Jx)  and  compared  for  maximum 
value. 

2.  fa2  >  fal  >  fa3.  In  this  case,  the  value  of  PiS'^Jx)  is  minimum  when  <5*2 
approaches  infinity  and  <5*3  =  0.  It  increases  with  smaller  5xZ  and  with  larger 

^13- 

Other  cases  are  possible,  but  according  to  the  first  one  above,  the  overall 
approach  for  maximizing  productivity  is  to  substitute  the  ratios  tx v/tx  1  and 
ixs/'tzi  f°r  each  candidate  in  the  formula  P  (S  ijx),  and  to  select  the  one  which 


yields  the  highest  value. 
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1-  >  *0.2  > 


2-  ^  ^al  ^  ^a3 


Fig.  16. 


3(SU*) 


-n. 


Arrows  indicate  the  directions  in  which/3  (S'  (Jar)  increases 
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APPENDIX  B 

Four  closed-form  Formulae  for  system  demand,  P(S  ^ Jx ),  for  multiprogram¬ 
ming  levels  ranging  from  two  to  five  are  presented. 


1.5  =  [a\ 

.  1  2DxDa 

P(S[JX)  =  -•  D'D'  -  £ 

jcK 


2.  S  =  \a,  b\ 


P(S'Ox) 


1 

3 


3DxDaDb  4  E  V>*Wbj  +  txj(Batbj  4  DbtaJ)] 
_ SfH _ 

DxDaDb  4-  E  [(Ac  )tajtbj  4  ^xj(Pa^bj  +  Pb^aj)] 

j<K 


3 .  S  =  {a,  b .  c  $ 

P(S  \Jx)  =  a/ 4rf3  where: 

a  =  4&zDaDbDc  4  E  i^x^ajhjtcs  +  2{^z  +  t^)' 

jzx 

(Pa  tbj  tcj  4  Pb  taj  tcj  4  Pctajtbj)  4  2tXj(DaDb  tCj  4  DaDr:ibj  4  DbDc  taj')'! 
P  —  Dx-Pa PbPc  +  E  K^Ar  +  6 txj)tajtbjtcj  4 

j±X 

t-xj  E  i^aj(Pb^c  4  ibk.tck )  4  tbj(DaDc  +  ^aJfc^c&)  4  tcj{Pa.Pb  4  fafc^fcjfc)]  4 

teif 


(Z?2.  4  2iXj)(DaibjtCj  -r  BbiajiCj  4  Pctajtbj)  } 
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4.  S  =  [a,  b,  c,  cZ } 

P (S  \J x )  =  a/bp  where: 

a  =  5 DxDaDbDcDd  +  ]T]  l®DxtajtbjtCjtdj  +  (3Z3x+4Z!ej)  ' 

j<uc 

(DaDbtCjtdj  -*■  D  aD  ctbjt  dj  DaDdtbjtcj  DbDctajtdj 
DbDdtajt  cj  +  DcDdtaitbj)  +  (^Dx+6tXj)(Datbjtcjtdj  + 

Db^a Dctajtbjidj  Ddtajtbjtrj ) 

3txj(DaDb  Dctdj  +  Da  D^D^tcj  +  D^DcD^t^  +  DaDcDdtbj)  + 

^xj  2  [(-^u tbj  **"  ^b^aj)^ck^dk  (Dctaj  ^a^cj)^bk^dk 
keJC 

i^a^dj  "*■  Ddl11j)lbjcLcjc  +  (Dbtcj  Dclbj)tajctdjc 
{Db^dj  "*■  Ddtbj)tajctc)c  C^c^dj  +  ^d^cj)^ak  ^bk  ]  + 

D x  2  {tajtbjtcJctdJc  ^  ^a;^c j^bk^ctk  +  ^aj^dj^Ok^ck)) 
fee* 


@  —  DxDaDbDcD d  +  2  \(^DX  24Za.y)Zayi5jZCjZ(£y  + 

jzK 

^xj[^aj^bj(DcDd  +  X  ^cfe^dfe)  +  ^aj^cj(DbDd  +  2  ^bk^dk)  + 

fee*  feeA' 

*a  jtdj(DbDc  +  X  tbktck)  +  ^bj^ej(DaDd  +  XI  + 

fee*  fee* 

thjtdj{Dn.Dr,  +  X  ^nktr.k)  +  tc.jtdj(DnDb  +  X  ^nfe^bfe)]  + 
fc e*  k  e* 

(2 Dx  +  SZ^j-  ){Datbjtcj  tdj  +■  Db  tajtcjtdj  ■*"  Dc  taj  tfrj  tdj  +  Ddtajtbjtcj) 

^x}L^aj{Dt)D0Dd  +  X  (.Db^ck^dk  ^ c^bk^dk  ^d^bk^ck  +  ^tbk^ck^dk))  + 

fee* 

tbj(DaDcDd  -*"  X  C^a^ck^dfc  +  ^c^ak  *dk  +  Ddtaktck  +  2*afe  *cfe  ^<ffe  ))  + 

feeA' 

hj(DaDbDd  +  X  (-Dahktdk  +  &b  t-dk  *dk  +  -d^akhk  +  ^akhk^dk))  + 


tr?j(DaDhDc  +  X  (- Qo.tbktck  +  Dbtaktck  +  D0ta^tbk  +  2Zafc Z6fc Zcfc))j  + 

i  fK 

Dx  [tajhj(DcDd  +  X  tcktdk)  +  tajtcj(DbDd  +  X  hitdk)  + 


ksK 


keX 


taj£dj(Db.Dc  4"  t’bk  t’Ck  )  £bj^CjDnDd  4"  thjidjQn^Dr<  4 

ifeeif 
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APPENDIX  C 


Theorem.  Under  FCFS  micro-scheduling,  given  service  set  5",  P(S! )  is  solely 


determined  by  the  mean  service  times  required  per  visit  to  a  processor  by  each 


job  in  S',  and  by  the  ratios  of  the  number  of  visits  each  job  in  5"  makes  to  each 


processor. 


The  proof  is  similar  to  the  proof  presented  in  Section  4.3. 

Lemma.  Given  the  service  set  S'^jx,  where  S  is  a  parameter  and  x  is  a  desig¬ 


nated  job,  P(S{Jx)  is  determined  by  the  mean  service  times  required  by  job  x 


per  visit  to  a  processor,  and  by  the  ratios  of  the  number  of  visits  job  x  makes  to 
each  processor. 

Proof.  Let  x  be  associated  with  a  vector  of  mean  service  demands  per  visit  to  a 
processor,  ft*  (0*1,...,  oxk),  and  with  a  vector  of  the  number  of  required  visits  to 
each  processor,  dx  =  (vx^...,vx]c).  The  lemma:  says  that  P(S\^)x)  is  determined 
by  vectors  frx  and  vbx,  where  ibx  -  {vjx\,...twxk),  vjxi  =  vZj/vxq  f°r  a^  K  and 
some  prcspRcified  processor  q,  qE  K. 

Dividing  both  the  numerator  and  the  denominator  in  the  first  term  of  Equa¬ 
tion  10  by  Vxq  results  in: 


E  w*$Vxs 

K 


P(s  u*)  =  C 


jeK 


As  the  first  term  above  is  determined  by  &x  and  by  zt>x,  it  remains  to  show  that 


so  is  Ri  (S  \jx ).  From  Equation  9,  after  dividing  the  numerator  and  the  denomi- 

i  *  X 


nator  of  its  second  term  by  vxq ,  it  follows  that: 


Ri  {S  l)x)  =  Dt  + 


2  vij<jxs 


2  vij  Taj  (•S’  O' x  i  ) 


a^S-i 


Ra(S  iJX-l) 


Applying  Equation  8  and  Little’s  Law: 


TaJ(S  \jx-i)  =  Vgjitraj  +  £  (Tcj 


ceS  '\Jx-i-a 


Tcj(Sjx-i-  a) 
KC(S  \Jx-i-a)  J 


Keeping  the  recursion  on  TCj(S  (jar—  i  —  a  )  until  all  classes  in  S1  but  one,  b,  are 
exhausted  results  in: 


jsK 

Hence,  the  recursive  formula  for  (5‘ljx)  is  determined  by  &x  and  ihx.  The 

i  *  x 

recursion  stops  when  only  two  classes,  b  and  x,  remain  in  the  service  set.  For 
the  last  two  classes  Equation  9  is  reduced  to: 


QE 


The  theorem  is  proven  by  applying  the  lemma,  in  turn,  for  each  job  x  in  5",  such 
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J.D.  Gorrie,  February  1971 
[M.A.Sc.  Thesis,  EE  1971] 

*  CSRG-4  DYLAN  USER'S  MANUAL 

P.E.  Bonzon,  March  1971 

CSRG-5  DIAL  -  A  PROGRAMMING  SYSTEM  FOR  INTERACTIVE  ALGEBRAIC  MANIPULATION 
Alan  C.M.  Brown  and  J.J.  Horning,  March  1971 

CSRG-6  ON  DEADLOCK  IN  COMPUTER  SYSTEMS 
Richard  C.  Holt,  April  1971 
[Ph.D.  Thesis,  Dept,  of  Computer  Science, 

Cornell  University,  1971] 


CSRG-7  THE  STAR-RING  SYSTEM  OF  LOOSELY  COUPLED  DIGITAL  DEVICES 
John  Neill  Thomas  Potvin,  August  1971 
[M.A.Sc.  Thesis,  EE  1971] 


*  CSRG-8  FILE  ORGANIZATION  AND  STRUCTURE 

G.M.  Stacey,  August  1971 

CSRG-9  DESIGN  STUDY  FOR  A  TWO-DIMENSIONAL  COMPUTER-ASSISTED 
ANIMATION  SYSTEM 
Kenneth  B.  Evans,  January  1972 
[M.Sc.  Thesis,  DCS,  1972] 

*  CSRG- 10  HOW  A  PROGRAMMING  LANGUAGE  IS  USED 

William  Gregg  Alexander,  February  1972 

[M.Sc.  Thesis,  DCS  1971;  Computer,  v.B,  n.  11.  November  1975] 

*  CSRG- 11  PROJECT  SUE  STATUS  REPORT 

J.W.  Atwood  (ed.),  April  1972 


+  Abbreviations: 

DCS  -  Department  of  Computer  Science,  University  of  Toronto 

EE  -  Department  oT  Electrical  Engineering,  University  of 

Toronto 
*  -  Out  of  print 
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*  CSRC-12  THRICE  DIMENSIONAL  DATA  DISPLAY  WITH  HIDDEN  LINE  REMOVAL 

Rupert  Hr  am  all,  April  1972 
[M.Sc.  Thesis,  DCS,  1971] 

-  CSRG-13  A  SYNTAX  DIRECTED  ERROR  RECOVERY  METHOD 

Lewis  R.  James,  May  1972 
[M.Sc.  Thesis,  DCS,  1972] 

CSRG-14  THE  USE  OF  SERVICE  TIME  DISTRIBUTIONS  IN  SCHEDULING 
Kenneth  C.  Sevcik,  May  1972 

[Ph.D.  Thesis,  Committee  on  Information  Sciences, 

University  of  Chicago,  1971;  JACM,  January  1974] 

CSRG-15  PROCESS  STRUCTURING 

J. J.  Horning  and  B.  Randell,  June  1972 
[ACM  Computing  Surveys,  March  1972] 

CSRG-16  OPTIMAL  PROCESSOR  SCHEDULING  WHEN  SERVICE  TIMES  ARE 

HYPEREXPONENTJALLY  DISTRIBUTED  AND  PREEMPTION  OVERHEAD 

IS  NOT  NEGLIGIBLE 

Kenneth  C.  Sevcik,  June  1972 

[Proceedings  of  the  Symposium  on  Computer-Communication, 
Networks  and  Teletraffic,  Polytechnic  Institute  of  Brooklyn,  1972] 

*  CSRG-17  PROGRAMMING  LANGUAGE  TRANSLATION  TECHNIQUES 

W.M.  McKeernan,  July  1972 

CSRG-1B  A  COMPARATIVE  ANALYSIS  OF  SEVERAL  DISK  SCHEDULING  ALGORITHMS 
C.J.M.  Turnbull,  September  1972 

CSRG-19  PROJECT  SUE  AS  A  LEARNING  EXPERIENCE 

K. C.  Sevcik  at  al,  September  1972 
[Proceedings  AFIPS  Fall  Joint  Computer  Conference, 
v.  41,  December  1972] 

*  CSRG-20  A  STUDY  OF  LANGUAGE  DIRECTED  COMPUTER  DESIGN 

David  B.  Wortman,  December  1972 

[Ph.D.  Thesis,  Computer  Science  Department, 

Stanford  University,  1972] 

CSRG-21  AN  APL  TERMINAL  APPROACH  TO  COMPUTER  MAPPING 
R.  Kvaternik,  December  1972 
[M.Sc.  Thesis,  DCS,  1972] 

-  CSRG-22  AN  IMPLEMENTATION  LANGUAGE  FOR  MINICOMPUTERS 

G.  G.  Kalmar,  January  1973 
[M.Sc.  Thesis,  DCS,  1972] 

CSRG-23  COMPILER  STRUCTURE 

W.M.  McKoeman,  January  1973 

[Proceedings  of  the  USA-Japnn  Computer  Conference,  1972] 
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*  CSRG-24 

CSRG-25 

*  CSRG-26 

*  CSRG-27 

*  CSRG-28 

*  CSRG-29 

*  CSRG-30 

* CSRG-31 

*  CSRG-32 

*  CSRG-33 

*  CSRG-34- 

*  CSRG-35 

*  CSRG-36 

*  CSRC-37 


AN  ANNOTATED  BIBLIOGRAPHY  ON  COMPUTER  PROGRAM 
ENGINEERING 

J.D.  Gannon  (ed.),  March  1973 

THE  INVESTIGATION  OF  SERVICE  TIME  DISTRIBUTIONS 
Eleanor  A.  Lester,  April  1973 
[M.Sc.  Thesis,  DCS,  1973] 

PSYCHOLOGICAL  COMPLEXITY  OF  COMPUTER  PROGRAMS: 

AN  INITIAL  EXPERIMENT 
Larry  Weissman,  August  1973 

STRUCTURED  SUBSETS  OF  THE  PL/'I  LANGUAGE 

Richard  C.  Holt  and  David  B.  Wortman,  October  1973 

ON  REDUCED  MATRIX  REPRESENTATION  OF  LR(k) 

PARSER  TABLES 

Marc  Louis  Joliat,  October  1973 

[Ph.D.  Thesis,  EE  1973] 

A  STUDENT  PROJECT  FOR  AN  OPERATING  SYSTEMS  COURSE 
B.  Czarnik  and  D.  Tsichritzis  (eds.),  November  1973 

A  PSEUDO-MACHINE  FOR  CODE  GENERATION 
Henry  John  Pasko,  December  1973 
[M.Sc.  Thesis,  DCS  1973] 

AN  ANNOTAED  BIBLIOGRAPHY  ON  COMPUTER  PROGRAM  ENGINEERING 
J.D.  Gannon  (ed.),  Second  Edition,  March  1974 

SCHEDULING  MULTIPLE  RESOURCE  COMPUTER  SYSTEMS 

E. D.  Lazowska,  May  1974 
[M.Sc.  Thesis,  DCS,  1974] 

AN  EDUCATIONAL  DATA  BASE  MANAGEMENT  SYSTEM 

F.  Lochovsky  and  D.  Tsichritzis,  May  1974 
[INFOR,  14  (3),  pp. 270-278,  1976] 

ALLOCATING  STORAGE  IN  HIERARCHICAL  DATA  BASES 
P.  Bernstein  and  D.  Tsichritzis,  May  1974 
[Information  Systems  Journal,  v.  1,  pp.  133-14  0] 

ON  IMPLEMENTATION  OF  RELATIONS 
D.  Tsichritzis,  May  1974- 

SIX  PL/I  COMPILERS 

D.B.  Wortman,  P.J.  Khaiat,  and  D.M.  Lasker,  August  1974 
[Software  Practice  and  Experience,  v.6,  n.3, 

Juiy-Sept.  1976] 


A  METHODOLOGY  FOR  STUDYING  THE  PSYCHOLOGICAL  COMPLEXITY 
OF  COMPUTER  PROGRAMS 


Laurence.  M.  Weissman,  August  1974 
|  Pli.D.  Thesis,  DCS,  1974  ] 
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*  CSRG-3B  AN  INVESTIGATION  OF  A  NEW  METHOD  OF  CONSTRUCTING  SOFTWARE 

David  M.  Lasker,  September  1974 
[M.Sc.  Thesis,  DCS,  1974] 

CSRG-39  AN  ALGEBRAIC  MODEL  FOR  STRING  PATTERNS 
Glenn  F.  Stewart,  September  1974 
[M.Sc.  Thesis,  DCS,  1974] 

*  CSRG-40  EDUCATIONAL  DATA  BASE  SYSTEM  USER’S  MANUAL 

J.  KlebanofT,  F.  Lochovskv,  A.  Rozitis,  and 

D.  Tsichritzis,  September  1974 

*  CSRG-41  NOTES  FROM  A  WORKSHOP  ON  THE  ATTAINMENT  OF 

RELIABLE  SOFTWARE 

David  B.  Wortman  (ed.),  September  1974 

*  CSRG-42  THE  PROJECT  SUE  SYSTEM  LANGUAGE  REFERENCE  MANUAL 

B.L.  Clark  and  F.J.B.  Ham,  September  1974 

*  CSRG-43  A  DATA  BASE  PROCESSOR 

E. A.  Ozkarahan,  S.A.  Schuster  and  K.C.  Smith, 

November  1974  [Proceedings  National  Computer 
Conference  1975,  v.44,  pp. 379-388] 

*  CSRG-44  MATCHING  PROGRAM  AND  DATA  REPRESENTATION  TO  A 

COMPUTING  ENVIRONMENT 
Eric  C.R.  Hehner,  Novemver  1974 
[Ph.D.  Thesis,  DCS,  1974] 

See  Computer,  Vol.9,  No. 9,  August  1976,  pp. 65-70. 

*  CSRG-45  THREE  APPROACHES  TO  RELIABLE  SOFTWARE;  LANGUAGE  DESIGN, 

DYADIC  SPECIFICATIONS,  COMPLEMENTARY  SEMANTICS 
J.E.  Donahue,  J.D.  Gannon,  J.V.  Guttag  and 
J.J.  Horning,  December  1974 

CSRG-46  THE  SYNTHESIS  OF  OPTIMAL  DECISION  TREES  FROM 
DECISION  TABLES 

Helmut  Schumacher,  December  1974 

[M.Sc.  Thesis,  DCS,  1974;  CACM,  v.19,  n.6,  June  1976] 

*  CSRG-47  LANGUAGE  DESIGN  TO  ENHANCE  PROGRAMMING  RELIABILITY 

John  D.  Gannon,  January  1975 
[Ph.D.  Thesis,  DCS,  1975] 

*  CSRG-48  DETERMINISTIC  LEFT  TO  RIGHT  PARSING 

Christopher  J.M.  Turnbull,  January  1975 
[Ph.D.  Thesis,  EE,  1974] 

*  CSRG-49  A  NETWORK  FRAMEWORK  FOR  RELATIONAL  IMPLEMENTATION 

D.  Tsichritzis,  February  1975  [in  Data  Base  Description, 

Dnnguo  and  Nijssnn  (eds.),  North  Holland  Publishing  Co.] 


CSRG-50  A  UNIFIED  APPROACH  TO  FUNCTIONAL  DEPENDENCIES 
AND  RELATIONS 

P.A.  Bernstein,  J.R.  Swenson  and  D.C.  Tsichrilzis 
February  1975  [Proceedings  of  the  ACM  SIGMOD 
Conference,  1975] 


CSRG-51  ZETA:  A  PROTOTYPE  RELATIONAL  DATA  BASE  MANAGEMENT  SYSTEM 
M.  Brodie  (ed).  February  1975  [Proceedings  Pacific  ACM 
Conference,  1975] 

CSRG-52  AUTOMATIC  GENERATION  OF  SYNTAX-REPAIRING  AND 
PARAGRAPHING  PARSERS 
David  T.  Barnard,  March  1975 
[M.Sc.  Thesis,  DCS.  1975] 


CSRG-53 


QUERY  EXECUTION  AND  INDEX  SELECTION  FOR  RELATIONAL 
DATA  BASES 

J.H.  Gilles  Farley  and  Stewart  A.  Schuster,  March  1975 


CSRG-54  AN  ANNOTATED  BIBLIOGRAPHY  ON  COMPUTER  PROGRAM 
ENGINEERING 

J.V.  Guttag  (ed.),  Third  Edition,  April  1975 

CSRG-55  STRUCTURED  SUBSETS  OF  THE  PL/1  LANGUAGE 

Richard  C.  Holt  and  David  B.  Wortman,  May  1975 


CSRG-56  FEATURES  OF  A  CONCEPTUAL  SCHEMA 

D.  Tsichritzis,  June  1975  [Proceedings  Very  Large 
Data  Base  Conference,  1975] 

CSRG-57  MERLIN:  TOWARDS  AN  IDEAL  PROGRAMMING  LANGUAGE 
Eric  C.R.  Hehner,  July  1975 

see  Acta  Informatica  Col.  10,  No. 3,  pp. 229-243,  1973 


C3RG-58  ON  THE  SEMANTICS  OF  THE  RELATIONAL  DATA  MODEL 
Hans  Albrecht  Schmid  and  J.  Richard  Swenson, 

July  1975  [Proceedings  of  the  ACM  SIGMOD  Conference,  1975] 

CSRG-59  THE  SPECIFICATION  AND  APPLICATION  TO  PROGRAMMING 
OF  ABSTRACT  DATA  TYPES 
John  V.  Guttag,  September  1975 
[Ph.D.  Thesis,  DCS,  1975] 

CSRG-60  NORMALIZATION  AND  FUNCTIONAL  DEPENDENCIES  IN  THE 
RELATIONAL  DATA  BASE  MODEL 
Phillip  Alan  Bernstein,  October  1975 
[Ph.D.  Thesis,  DCS,  1975] 

CSRG-61  LSL:  A  LINK  AND  SELECTION  LANGUAGE 

D.  Tsichritzis,  November  1975  [Proceedings  ACM 
SIGMOD  Conference,  1975] 
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*  CSRG-62 


CSRG-63 


CSRG-64 


CSRG-65 


CSRG-66 


CSRG-67 


*  CSRG-68 


CSRG-69 


*  CSRG-70 


*  CSRG-71 


*  CSRG-72 


COMPLEMENTARY  DEFINITIONS  OF  PROGRAMMING  LANGUAGE 
SEMANTICS 

James  E.  Donahue,  November  1975 
[Ph.D.  Thesis,  DCS,  1975] 

AN  EXPERIMENTAL  EVALUATION  OF  CHESS  PLAYING  HEURISTICS 
Lazio  Sugar,  December  1975 
[M.Se.  Thesis.  DCS,  1975] 

A  VIRTUAL  MEMORY  SYSTEM  FOR  A  RELATIONAL  ASSOCIATIVE 
PROCESSOR 

S.A.  Schuster,  E.A.  Ozkarahan,  and  K.C.  Smith, 

February  1976  [Proceedings  National  Computer 
Conference  1976,  v.45,  pp. 855-862] 

PERFORMANCE  EVALUATION  OF  A  RELATIONAL,  ASSOCIATIVE 
PROCESSOR 

E.A.  Ozkarahan,  S.A.  Schuster,  and  K.C.  Sevcik, 

February  1976  [ACM  Transactions  on  Database 
Systems,  v.  1,  n:4,  December  1976] 

EDITING  COMPUTER  ANIMATED  FILM 
Michael  D.  Tilson,  February  1976 
[M.Sc.  Thesis,  DCS,  1975] 


A  DIAGRAMMATIC  APPROACH  TO  PROGRAMMING  LANGUAGE 
SEMANTICS 

James  R.  Cordy,  March  1976 
[M.Sc.  Thesis,  DCS,  1976] 

A  SYNTHETIC  ENGLISH  QUERY  LANGUAGE  FOR  A  RELATIONAL 
ASSOCIATIVE  PROCESSOR 

L.  Kersehberg,  E.A.  Ozkarahan,  and  J.E.S.  Pacheco, 

April  1976 


AN  ANNOTATED  BIBLIOGRAPHY  ON  COMPUTER  PROGRAM 
ENGINEERING 

D.  Barnard  and  D.  Thompson  (eds.),  Fourth  Edition, 
May  1976 


A  TAXONOMY  OF  DATA  MODELS 

L.  Kersehberg,  A.  Klug,  and  D.Tsichritzis,  May  1976 
[Proceedings  Very  Large  Data  Base  Conference,  1976] 


OPTIMIZATION  FEATURES  FOR  THE  ARCHITECTURE  OF  A 
DATA  BASE  MACHINE 


E.A.  Ozkarahan  and  K.C.  Sevcik,  May  1976 

[ACM  Transactions  of  Database  Systems,  v.2,  n.4,  December  1977] 


THE  RELATIONAL  DATA  BASE  SYSTEM  OMEGA  -  PROGRESS  REPORT 
II. A,  Schmid  (ed.),  P.A.  Bernstein  (ed.),  B.  Arlow, 

R.  Baker  ami  S.  Po/.gaj,  July  1976 
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CSRG-73  AN  ALGORITHMIC  APPROACH  TO  NORMALIZATION  OK 
RELATIONAL  DATA  RASE  SCHEMAS 
P.A.  Bernstein  and  C.  Beeri,  September  1976 

*  CSRG-74  A  HIGH-LEVEL  MACHINE-ORIENTED  ASSEMBLER  LANGUAGE 
FOR  A  DATA  BASE  MACHINE 

E.A.  Ozkarahan  and  S.A.  Schuster,  October  1976 

CSRG-75  DO  CONSIDERED  OD:  A  CONTRIBUTION  TO  THE  PROGRAMMING 
CALCULUS 

Eric  C.R.  Hehner,  November  1976 
Acta  Informatica  to  appear  1979 

CSRG-76  SOFTWARE  HUT:  A  COMPUTER  PROGRAM  ENGINEERING 
PROJECT  IN  THE  FORM  OF  A  GAME 
J.J.  Horning  and  D.B.  Wortman,  November  1976 

[IEEE  Transactions  on  Software  Engineering,  v.SE-3,  n.4,  July  1977] 

CSRG-77  A  SHORT  STUDY  OF  PROGRAM  AND  MEMORY  POLICY  BEHAVIOUR 
G.  Scott  Graham,  January  1977 

CSRG-78  A  PANACHE  OF  DBMS  IDEAS 

D.  Tsichritzis  (ed.),  February  1977 

CSRG-79  THE  DESIGN  AND  IMPLEMENTATION  OF  AN  ADVANCED  LAI.R 
PARSE  TABLE  CONSTRUCTOR 
David  H.  Thompson,  April  1977 
[M.Sc.  Thesis,  DCS,  1976] 

CSRG-80  AN  ANNOTATED  BIBLIOGRAPHY  ON  COMPUTER  PROGRAM 
ENGINEERING 

D.  Barnard  (ed.),  Fifth  Edition,  May  1977 

CSRG-81  PROGRAMMING  METHODOLOGY:  AN  ANNOTATED  BIBLIOGRAPHY 
FOR  1FIP  WORKING  GROUP  2.3 

Sol  J.  Greenspan  and  J.J.  Horning  (eds.),  First  Edition,  May  1977 
CSRG-82  NOTES  ON  EUCLID 

edited  bv  W.  David  Elliot  and  David  T.  Barnard,  August  1977 

CSRG-83  TOPICS  IN  QUEUEING  NETWORK  MODELING 
edited  by  G.  Scott  Graham,  July  1977 

CSRG-84  TOWARD  PROGRAM  ILLUSTRATION 

Edward  Yarwood,  September  1977 
[M.Sc.  Thesis,  DCS,  1974] 

CSRG-85  CHARACTERIZING  SERVICE  TIME  AND  RESPONSE  TIME 

DISTRIBUTIONS  IN  QUEUEING  NETWORK  MODELS  OF  COMPUTER 
SYSTEMS 

Edward  D.  Lazowska,  September  1977 
[Ph.D.  Thesis,  DCS,  1977| 
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CSRG-86  MEASUREMENTS  OF  COMPUTER  SYSTEMS  FOR  QUEUEINC 
NETWORK  MODELS 
Martin  G.  Kienzle,  October  1977 

[M.Sc.  Thesis,  DCS,  1977;  Proc.  Int.  Symp.  on  Modelling  and  Performance 
Evaluation  of  Computer  Systems,  Vienna,  1979] 

CSRG-87  ’OLGA'  LANGUAGE  REFERENCE  MANUAL 

B.  Abourbih,  H.  Trickey,  D.M.  Lewis,  E.S.  Lee, 

P.I.P.  Boulton,  November  1977 

CSRG-88  USING  A  GRAMMATICAL  FORMALISM  AS  A  PROGRAMMING  LANGUAGE 
Brad  A.  Silverberg,  January  1978 
[M.Sc.  Thesis,  DCS,  1978] 

CSRG-89  ON  THE  IMPLEMENTATION  OF  RELATIONS:  A  KEY  TO  EFFICIENCY 
Joachim  W.  Schmidt.  January  1978 

CSRG-90  DATA  BASE  MANAGEMENT  SYSTEM  USER  PERFORMANCE 
Frederick  H.  Lochovsky,  April  1978 
[Pb.D.  Thesis,  DCS,  1978] 

CSRG-91  SPECIFICATION  AND  VERIFICATION  OF  DATA  BASE 
SEMANTIC  INTEGRITY 
Michael  Lawrence  Brodie,  April  1978 
[Ph.D.  Thesis,  DCS,  1978] 

CSRG-92  STRUCTURED  SOUND  SYNTHESIS  PROJECT  (SSSP): 

AN  INTRODUCTION 

by  William  Buxton,  Guy  Fedorkow,  with  Ronald  Baecker, 

Gustav  Ciamaga,  Leslie  Mezei  and  K.C.  Smith,  June  1978 

CSRG-93  A  DEVICE-INDEPENDENT, GENERAL-PURPOSE  GRAPHICS  SYSTEM 
IN  A  MINICOMPUTER  TIME-SHARING  ENVIRONMENT 
William  T.  Reeves,  August  1978 
[M.Sc.  Thesis,  DCS,  1976] 

CSRG-94  ON  THE  AXIOMATIC  VERIFICATION  OF 
CONCURRENT  ALGORITHMS 
Christian  Lengauer,  August  1978 
[M.Sc.  Thesis,  DCS,  1978] 

CSRG-95  PISA:  A  PROGRAMMING  SYSTEM  FOR  INTERACTIVE 
PRODUCTION  OF  APPLICATION  SOFTWARE 
Rudolf  Marty,  August  1978 

CSRG-96  ADAPTIVE  MICROPROGRAMMING  AND  PROCESSOR  MODELING 
Walter  G.  Rosocha 
[Ph.D.  Thesis,  EE,  August  1978] 

CSRG-97  DESIGN  ISSUES  IN  THE  FOUNDATION  OF  A  COMPUTER-BASED 
TOOL  FOR  MUSIC  COMPOSITION 
William  Buxton 

[M.Sc.  Thesis,  CSRG,  October  1978] 
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CSRG-98  THEORY  OF  DATABASE  MAPPINGS 
Anthony  C.  Klug 

[Ph.D.  Thesis,  DCS,  December  1978] 

CSRG-99  HIERARCHICAL  COROUTINES:  A  MECHANISM  FOR  IMPROVED 
PROGRAM  STRUCTURE 
Leonard  I.  Vanek,  February  1979 

CSRG-100  TOPICS  IN  PERFORMANCE  EVALUATION 
G.  Scott  Graham  (ed.)t  July  1979 

CSRG-101  A  PANACHE  OF  DBMS  IDEAS  II 

F.H.  Lochovsky  (ed.),  May  1979 

CSRG-102  A  SIMPLE  SET  THEORY  FOR  COMPUTING  SCIENCE 
Eric  C.R.  Hehner,  May  1979 

CSRG-103  THE  CENTRALIZED  ALGORITHM  IN  DISTRIBUTED  SYSTEMS 
Ernest  J.H.  Chang 
[Ph.D.  Thesis,  DCS,  July  1979] 

CSRG-104  ELIMINATING  THE  VARIABLE  FROM  DIJKSTRA’S 
MINI-LANGUAGE 
D.  Hugh  Redelmeier,  July  1979 

CSRG-105  A  LANGUAGE  FACILITY  FOR  DESIGNING  INTERACTIVE 
DATABASE-INTENSIVE  APPLICATIONS 
John  Mylopoulos,  Philip  A.  Bernstein,  Harry  K.T.  Wong, 
July  1979 

CSRG-106  ON  APPROXIMATE  SOLUTION  TECHNIQUES  FOR 

QUEUEING  NETWORK  MODELS  OF  COMPUTER  SYSTEMS 
Satish  Kumar  Tripathi,  July  1979 

CSRG-107  A  FRAMEWORK  FOR  VISUAL  MOTION  UNDERSTANDING 
John  K.  Tsotsos,  John  Mylopoulos,  H.  Dominic  Cowey 
Steven  W.  Zucker,  DCS,  June  1979 

CSRG-108  DIALOGUE  ORGANIZATION  AND  STRUCTURE  FOR 
INTERACTIVE  INFORMATION  SYSTEMS 
John  Leonard  Barron 
[M.Sc.  Thesis,  DCS,  1980] 

CSRG-109  A  UNIFYING  MODEL  OF  PHYSICAL  DATABASES 
D.S.  Batory,  C.C.  Gotlieb,  April  1980 

CSRG-110  OPTIMAL  FILE  DESIGNS  AND  REORGANIZATION  POINTS 
D.S.  Batory,  April  1980 

CSRG-1 1 1  A  PANACHE  OF  DBMS  IDEAS  III 
D.  Tsichril./.is  (ed.),  April  1980 
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CSRG-1 12  TOPICS  IN  PSN  -  II:  EXCEPTIONAL  CONDITION 

HANDLING  IN  PSN;  REPRESENTING  PROGRAMS  IN  PSN; 
CONTENTS  IN  PSN 

Yves  Lesperance.  Byran  M.  Kramer,  Peter  F.  Schneider 
April,  1980.. 

CSRG-1 13  SYSTEM-ORIENTED  MACRO-SCHEDULING 
C.C.  Gotlieb.  and  A.  Schonbach 
May  I960 

CSRG-1 14  A  FRAMEWORK  FOR  VISUAL  MOTION  UNDERSTANDING 
John  Konstantine  Tsotsos 
[Ph.D.  Thesis,  DCS,  June  1980] 

CSRG-1 15  SPECIFICATION  OF  CONCURRENT  EUCLID 
James  R.  Cordy  and  Richard  C.  Holt 
July  1980  ' 

CSRG-1 16  THE  REPRESENTATION  OF  PROGRAMS  IN  THE 

PROCEDURAL  SEMANTIC  NETWORK  FORMALISM 

Bryan  M.  Kramer 

[M.Sc.  Thesis,  DCS,  1980] 

CSRG-1 17  CONTEXT-FREE  GRAMMARS  AND  DERIVATION  TREES  AS 
PROGRAMMING  TOOLS 
Volker  Linnemann 
September  1980 

\ 

CSRG-1 18  S/SL:  SYNTAX/SEMANTIC  LANGUAGE 
INTRODUCTION  AND  SPECIFICATION 
R.C.  Holt,  JiR.  Cordy,  D.B.  Wortman 
CSRG,  September  1980 

CSRG-1 19  PT:  A  PASCAL  SUBSET 
Alan  Rosselet 

[M.Sc.  Thesis,  DCS,  October  1980] 

CSRG- 120  PTED:  A  STANDARD  PASCAL  TEXT  EDITOR  BASED  ON 
THE  KERNIGHAN  AND  PLAUCER  DESIGN 
Ken  Newman,  DCS 
October  I960 

CSRG-121  TERMINAL  CONTEXT' GRAMMARS 
Howard  W.  Trickey 
[M.Sc.  Thesis,  EE,  September  1980] 

CSRG-122  THE  APPROXIMATE  SOLUTION  OF  LARGE  QUEUEING 
NETWORK  MODELS 
John  Zahorjan 

[Ph.D.  Thesis,  DCS,  August  1980]  ?  ; 
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CSRG-123  A  FORMAL  TREATMENT  OF  IMPERFECT  INFORMATION 
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